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ABSTRACT 


The  objective  of  this  research  effort  is  to  design  and  implement  a 
laboratory  and  establish  a  research  method  focused  on  scientific  evaluation  of 
human  factors  considerations  for  military  individual  equipment  under  both 
laboratory  and  field  conditions.  This  integrated  approach  for  laboratory  and  field 
conditions  is  the  first  of  its  kind  for  military  human  factors  research,  enabling  an 
unparalleled  degree  of  scientific  rigor  in  the  collection  of  empirical  human  factors 
data.  This  effort  includes:  1)  a  state-of-the-art  usability  laboratory  designed 
specifically  for  quantitatively  evaluating  military  individual  equipment;  2)  a 
rugged,  embarkable,  fully  self-contained,  portable  usability  laboratory  for  field 
research  in  military  environments;  3)  a  codified  manual  for  using  the  two  main 
configurations  (stationary  and  portable)  of  the  usability  laboratory,  written  for  the 
beginning  usability  researcher;  4)  a  set  of  validated  procedures  for  applying 
sound  human  factors  principles,  and  traditional  and  non-parametric  statistics  to 
the  specific  problem  of  usability  testing  of  military  individual  equipment;  5)  a 
proof-of-concept  practical  application  of  the  laboratory  and  procedures  to  a 
specific  problem,  namely  the  usability  testing  of  ruggedized  personal  digital 
assistants  (RPDAs)  designed  for  United  States  Special  Forces  operations. 
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EXECUTIVE  SUMMARY 


End  users  bear  the  burden  of  poor  design,  and  persuading  users  to  learn 
and  use  a  system’s  capabilities  goes  beyond  mere  functional  effectiveness. 
There  is  a  human  factors  gap  in  the  Test  and  Evaluation  process,  and  a  need 
within  the  Department  of  Defense  (DoD)  for  a  laboratory  and  a  reliable  research 
methodology  to  perform  scientifically  rigorous  human  factors  evaluations  that 
goes  beyond  subjective  observations  and  survey  data.  Rather  than  "next-bench" 
testing  between  technical  experts,  true  usability  testing  should  involve 
representative  participants  under  realistic  conditions. 

The  objective  of  this  research  effort  is  to  design  and  implement  a 
laboratory  and  establish  a  research  method  focused  on  scientific  evaluation  of 
human  factors  considerations  for  military  individual  equipment  under  both 
laboratory  and  field  conditions.  This  integrated  approach  will  represent  the  first  of 
its  kind  for  academic  and  military  human  factors  research,  enabling  an 
unparalleled  degree  of  scientific  rigor  in  the  collection  of  empirical  human  factors 
data.  This  effort  includes:  1)  a  state-of-the-art,  well-tested,  and  professional 
usability  laboratory  designed  specifically  for  quantitatively  evaluating  military 
individual  equipment;  2)  a  rugged,  embarkable,  fully  self-contained  portable 
usability  laboratory  for  field  research  in  all  military  environments;  3)  a  codified 
manual  for  using  the  two  main  configurations  (stationary  and  portable)  of  the 
usability  laboratory,  written  for  the  beginning  usability  researcher;  4)  a  set  of 
validated  procedures  for  applying  sound  human  factors  principles,  and  traditional 
and  non-parametric  statistics  to  the  specific  problem  of  usability  testing  of  military 
individual  equipment;  5)  a  proof-of-concept  practical  application  of  the  laboratory 
and  procedures  to  a  specific  problem,  namely  the  usability  testing  of  ruggedized 
personal  digital  assistants  (RPDAs)  designed  for  United  States  Special  Forces 
operations. 

In  March  of  2004,  the  Special  Operations  Command  (SOCOM)  requested 
a  human  factors  evaluation  of  the  RPDAs  they  have  fielded  in  Iraq  and 
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Afghanistan.  At  that  time,  the  DoD  did  not  have  a  facility  equipped  specifically  to 
test  military  individual  equipment  in  the  manner  needed  for  comprehensive 
software  and  hardware  usability  testing.  Consequently,  a  behavioral  observation 
laboratory,  the  Applied  Warfighter  Ergonomics  (AWE)  Center,  was  designed  and 
constructed  for  this  purpose.  Since  then,  the  AWE  Center  has  generated  serious 
interest  from  a  number  of  different  DoD  and  federal  agencies.  It  will  also  serve  as 
the  basis  for  much  future  research  at  NPS.  The  equipment  has  been  modified  at 
NPS  with  battery  packs  so  that  a  researcher  can  follow  a  user  into  the  field, 
recording  how  users  operate  the  equipment  and  the  context  with  which  it  is 
employed.  All  the  necessary  equipment  fits  into  three  waterproof  transit  cases,  so 
that  an  actual  field  study  could  be  conducted  to  validate  the  results  obtained  from 
a  controlled  laboratory  experiment. 

Digital  videos  recorded  in  the  laboratory  or  in  the  field  are  manually  coded 
into  data  files  using  the  Noldus  Observer  software.  The  technical  details  of  the 
process  have  been  written  into  a  Users  Guide,  which  distills  hundreds  of  pages 
of  documentation  and  usability  references  into  a  focused,  stand-alone  tutorial. 
The  developer  of  the  software  is  using  our  lab  as  a  model  on  their  website. 

This  thesis  details  an  experimental  design,  data  collection  method, 
analytical  procedure,  lessons  learned  and  recommendations  for  future 
researchers.  Both  qualitative  and  quantitative  means  are  employed  to  provide 
human  factors  usability  testing. 

To  validate  the  laboratory  and  the  procedure,  a  small  case  study 
comparing  three  different  RPDAs  was  conducted.  The  case  study  was  a  proof-of- 
concept  and  was  not  expected  to  be  conclusive.  Soldiers  and  Marines  at  the 
Defense  Language  Institute  were  brought  to  NPS  in  small  groups.  Participants 
were  given  a  pre-operation  survey  with  45  questions  that  attempt  to  capture  their 
technology  experience  and  military  background.  The  participants  went  through  a 
30-minute  tactical  scenario,  during  which  they  performed  various 
communications  and  navigation  tasks  while  equipped  with  body  armor  and  rifles. 
The  scenario  was  scripted  to  drive  the  participants  to  perform  the  tactical  tasks 


using  conventional  means  (such  as  with  a  paper  map),  in  order  to  provide  a 
baseline  measure  of  their  abilities.  Later  in  the  scenario,  they  performed  similar 
tasks  using  the  RPDA.  Quantitative  measurements  such  as  task  time,  tactical 
posture,  error  rate,  and  category  of  errors  were  recorded  while  users  employed 
both  conventional  and  RPDA  means  to  perform  the  tasks.  When  quantitative 
results  were  not  conclusive,  recommendations  were  possible  based  on 
qualitative  observations  in  the  lab  and  the  field. 

Inter-rater  reliability  statistics  verified  that  the  behavioral  measurements 
were  reliably  coded  using  well-defined  criteria.  We  compare  the  task  times 
assigned  by  different  researchers  on  the  same  video  observation  file  and 
compute  a  confusion  matrix,  index  of  concordance,  and  Cohen's  k. 

Quantifying  individual  differences  between  participants  reduces  the  noise 
in  the  measurements,  and  reduces  our  required  sample  size.  Testable  measures 
of  participant  effect,  like  the  baseline  tasks,  are  preferable  to  surveys,  which  are 
prone  to  a  variety  of  errors.  With  a  properly  constructed  experiment,  we  can 
justify  quantitative  analysis  of  human  factors  data. 

Parametric  and  non-parametric  statistics  were  used  to  compare  devices  in 
the  case  study.  High  participant  attrition  from  the  study  limited  the  extent  to  which 
rigorous  statistical  analysis  could  be  performed  on  the  data.  Some  users  voiced 
strong  opinions  about  the  devices  in  their  post-operation  surveys.  This  research 
has  shown  that  perception  can  influence  the  adoption  of  innovations  more 
strongly  than  quantitative  performance. 

This  thesis  fills  the  human  factors  gap  in  the  T&E  process.  There  are 
Special  Forces  operators  and  Marines  in  Iraq  and  Afghanistan  using  all  of  these 
devices  today.  Results  from  this  work  and  from  research  using  this  laboratory  will 
be  applied  by  SOCOM  to  improve  current  and  future  RPDAs.  This  laboratory  will 
make  the  Naval  Postgraduate  School  the  premiere  military  individual  equipment 
testing  site  in  the  country. 
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I.  INTRODUCTION 


A.  THESIS  ORGANIZATION 

This  research  effort  began  as  a  human  factors  study  of  military  ruggedized 
personal  digital  assistants  within  the  Naval  Postgraduate  School's  Tactical 
Network  Topology  project  for  the  United  States  Army  Special  Operations 
Command.  This  thesis  is  organized  into  five  major  parts.  After  the  introductory 
material,  Part  II:  Design  and  Implementation  describes  the  technical  details  of  the 
laboratory  designed  and  constructed  to  facilitate  the  study.  Part  III:  Approach 
describes  a  general  approach  specifically  for  conducting  a  usability  analysis  of 
military  individual  equipment  The  case  study  reported  in  Part  IV  serves  as  a 
proof-of-concept  for  the  laboratory  and  the  approach,  and  is  a  complete  study  in 
its  own  right.  The  final  conclusions  and  recommendations,  Part  V,  discuss  the 
future  of  this  laboratory,  its  applicability  to  other  problems,  and  ways  to  build  on 
this  groundwork  to  facilitate  rapid,  effective  usability  research  of  other  military 
individual  equipment.  The  appendices  contain  the  data  collected  during  the  case 
study  and  other  relevant  resources  and  can  be  used  as  a  guide  for  future  data 
collection  efforts. 

B.  PROBLEM  STATEMENT 

1.  Description 

The  successful  development  of  an  innovation  from  concept  to  an 
established  product  is  closely  linked  to  the  user's  ability  and  desire  to  employ  the 
product.  Developers  who  are  too  focused  on  the  underlying  technical  capabilities 
of  their  inventions  may  be  surprised  that  their  products  are  not  adopted  by  their 
targeted  users.  Apple  Computer  Fellow  Alan  Kay  tells  us  that  to  users,  “the 
interface  is  the  application,”  and  features  that  are  not  apparent,  do  not  exist. 

Traditional  human  factors  research,  including  usability  analysis,  often 
relies  on  subjective  data,  such  as  surveys  and  free-form  user  feedback.  These 

data  can  be  difficult  to  reproduce  and  be  prone  to  errors  due  to  user  bias  and 
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distorted  recollection.  Military  equipment  places  even  greater  demands  on  the 
usability  analyst.  Military  equipment  is  rarely  designed  for,  or  even  appropriately 
evaluated  in,  a  controlled  laboratory  environment.  Both  the  physical  and 
psychological  environments  in  which  military  equipment  is  employed  have 
impacts  on  the  human  factors  considerations  of  that  equipment. 

Military  individual  equipment  is  that  equipment  that  is  normally  carried  and 
employed  by  a  single  person  in  combat.  Examples  include:  personal  weapons, 
body  armor  and  protective  gear,  load  bearing  equipment,  navigation  aids,  and 
portable  communications  equipment.  Individual  equipment  is  relatively 
inexpensive  on  a  per-item  basis.  Unlike  the  research  and  development  programs 
for  larger  and  more  expensive  military  items,  individual  equipment  items  may  not 
be  subjected  to  extensive  human  factors  testing  or  systems  integration  with  the 
“whole  system”  encompassing  everything  a  ground  combat  Soldier  or  Marine 
may  typically  carry  with  them  into  battle.  Equipment  deficiencies  and 
incompatibilities  may  not  be  noted  until  the  equipment  is  actually  fielded. 

The  main  challenges  to  performing  usability  analysis  on  military  individual 
equipment  are  in  apparent  diametric  opposition: 

•  To  be  scientifically  valid,  the  evaluation  should  be 
repeatable,  comparable  between  systems,  and  ideally,  yield 
statistically  rigorous  results. 

•  To  have  predictive  and  inferential  relevance,  the  equipment 
should  be  tested  by  representative  users  under  realistic 
conditions. 

The  demand  for  validity  might  lead  us  to  perform  a  controlled  laboratory 
study  using  convenient  participants,  such  as  students.  The  compromises  in 
realism  to  capture  accurate  data  and  achieve  statistically  supportable  results  may 
lead  to  a  test  so  artificial  as  to  be  pointless.  Pursuing  the  goal  of  relevance, 
perhaps  by  conducting  a  field  exercise  using  real  Soldiers  and  Marines  in  a 
tactical  scenario,  results  in  a  test  that  is  not  repeatable,  with  many  variables 
being  uncontrolled,  confounded,  and  even  unidentified.  Highly  subjective  after¬ 
action  reviews  with  operators  are  subject  to  unintentionally  distorted  recollection 
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and  individual  biases.  Such  data  are  also  difficult  to  compare  across  different 
training  scenarios 

Military  individual  equipment  that  is  usable  and  appealing  to  users  will 
increase  the  rates  of  adoption  and  effective  employment.  This  thesis  proposes  a 
method  for  scientifically  evaluating  the  usability  and  other  human  factors 
considerations  of  modern  military  individual  equipment  under  both  laboratory  and 
field  conditions. 

2.  Background 

Various  agencies  within  the  Department  of  Defense  are  currently 
developing  and  fielding  a  number  of  handheld  computing  devices  known 
collectively  as  Ruggedized  Personal  Digital  Assistants  (RPDAs).  These  devices 
are  similar  to,  and  often  based  on,  the  commercial  personal  digital  assistants 
(PDAs)  known  as  "Palm  Pilots",  "Pocket  PCs",  or  "Windows  CE"  devices.  Civilian 
PDAs,  introduced  by  Apple  in  1993  and  popularized  by  Palm  in  1996,  introduced 
a  new  paradigm  in  computing  and  mobile  information  accessibility,  and  new 
human  interface  challenges  to  designers.  Military  handheld  computers  have 
unique  human  factors  considerations.  Civilian  PDA  interfaces  are  not  designed  to 
be  used  while  the  operator:  1)  simultaneously  attends  to  mental  and  physical 
external  stimuli  such  as  enemy  activity,  2)  wears  and  employs  other  equipment 
and  weapons  which  may  also  require  the  use  of  both  hands  and  eyes,  or  3) 
operates  under  environmental  extremes,  such  as  wearing  cold-weather  gloves. 

As  with  any  new  technology,  there  is  the  challenge  of  encouraging  and 
facilitating  its  adoption.  The  "diffusion  of  innovation"  of  electronic  devices  that 
supplant  existing  (manual)  systems  or  provide  novel  capabilities  can  be  slow 
when  users  have  little  experience  with  such  devices,  have  low  confidence  in  the 
electronic  device,  or  have  a  mission-critical  or  life-and-death  requirement  for 
reliability. 

Decades  of  human  factors  research  have  produced  guidelines  for  cockpit 
design,  control  panel  layouts,  and  websites,  among  other  interfaces.  Currently, 
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there  is  not  a  well-defined  methodology  to  evaluate  the  human  factors  issues  or 
adoption  strategies  pertinent  to  military  RPDAs. 

3.  Rationale  for  Conducting  Study 

Military  RPDAs  are  at  the  cutting  edge  of  human-machine  interface 
technology.  Human  factors  considerations  are  currently  being  accounted  for  in  an 
ad-hoc  and  "gut-instinct"  manner,  if  at  all.  This  study  will  produce  a  rigorous 
method  for  examining  the  human  factors  considerations  of  military  RPDAs.  The 
results  of  this  study  will  directly  influence  the  ongoing  development  and  fielding 
procedures  of  Special  Operation  Command's  current  military  RPDA.  Other 
organizations  will  be  able  to  use  this  research  to  better  design  and  field  military 
RPDAs  now  and  in  the  future. 

4.  Research  Questions 

What  human  factors  considerations  are  generic  to  handheld  devices  under 
normal  civilian  operating  conditions? 

What  human  factors  considerations  are  specific  to  handheld  computing 
equipment  used  by  military  personnel? 

How  can  these  human  factors  issues  be  tested  and  evaluated  in  a 
laboratory  environment,  and  what  are  the  drawbacks  of  laboratory  testing  and 
evaluation? 

How  can  these  human  factors  issues  be  tested  and  evaluated  in  a  field 
environment,  and  what  are  the  drawbacks  of  field  testing  and  evaluation? 

What  measures  of  performance  and  measures  of  effectiveness  should  be 
applied  to  the  human  interface  of  handheld  military  electronic  devices? 

What  factors  do  military  personnel  consider  during  the  adoption  of 
handheld  electronic  devices?  How  can  a  fielding  plan  address  these  factors  in 
order  to  facilitate  the  successful  training  and  employment  of  these  devices? 
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What  diffusion  of  innovation  considerations  should  be  incorporated  into 
the  research  and  development  of  military  personal  electronic  devices? 

C.  LITERATURE  REVIEW 

Several  fields  of  research  are  most  pertinent  to  this  study:  human- 
computer  interaction,  usability  testing,  and  the  study  of  the  diffusion  of 
innovations. 

1.  The  Diffusion  of  Innovations 

Everett  Rogers's  Diffusion  of  Innovations  (2003),  in  its  fifth  edition  since 
the  original  was  published  in  1962,  is  widely  recognized  as  the  foundation  work  in 
the  field.  His  basic  approach  and  specific  definitions  are  cited  in  virtually  all 
diffusion  and  adoption  literature.  Rogers  (2003)  defines  diffusion: 

The  main  elements  in  the  diffusion  of  new  ideas  are:  (1)  an 

innovation  (2)  that  is  communicated  through  certain  channels  (3) 

over  time  (4)  among  the  members  of  a  social  system. 

In  a  number  of  engaging  case  studies,  he  illustrates  the  ranges  and 
implications  of  innovations,  communication  channels,  time,  and  social  systems 
within  his  framework.  Rogers  identifies  five  categories  of  adopters  as  having 
distinct  characteristics  and  motivations  with  respect  to  innovation:  innovators, 
early  adopters,  the  early  majority,  the  skeptical  late  majority,  and  the  traditionalist 
laggards.  He  uses  these  and  other  specific  generalizations  to  draw  parallels 
between  case  studies  of  subjects  as  diverse  as  chicken  farmers  in  Africa,  doctors 
in  Illinois,  and  cell  phone  users  in  Finland.  The  first  edition  of  Diffusion  of 
Innovations  (Rogers,  1962)  introduced  the  classic  "S-shaped  curve  of  adoption" 
(Figure  1).  Through  a  series  of  extensive  and  rigorous  experiments,  Rogers 
demonstrated  that  the  cumulative  percentage  of  individual  adopters  of 
innovations  (ideas,  new  technologies,  or  social  practices)  follows  a  normal 
probability  distribution  under  a  wide  variety  of  circumstances.  Rogers  then 
demonstrates  that  the  cumulative  distribution  of  adopters  generally  follows  the 
same  S-shaped  curve  for  organizations  adopting  innovations,  as  well. 
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Subsequent  studies  have  established  that  the  top  official,  head  of  household,  or 
other  nominal  leader  of  the  organization  is  not  necessarily  the  decision  maker  in 
the  adoption  process.  Identifying  a  specific  individual  responsible  for  adoption 
decisions  is  often  complex  or  even  fruitless. 

Figure  7-1.  The  Number  of  New  Adopters  Each  Year,  and  the 
Cumulative  Number  of  Adopters,  of  Hybrid  Seed  Com  in  Two  Iowa 
Communities 


Year 

The  cumulative  number  of  adopters  of  hybrid  seed  com  approaches  an 
S-shaped  curve  over  time,  while  the  frequency  distribution  ol  the  number 
of  mean  adopters  per  year  approaches  a  normal,  bell-shaped  curve.  Early 
in  the  diffusion  process,  relatively  few  individuals  adopt  in  each  time 
period.  Gradually,  however,  the  rate  of  adoption  speeds  up  until  all  (or 
almost  all)  members  of  the  system  adopt  the  innovation. 

Source:  Based  on  Ryan  and  Gross  (1943). 

Figure  1.  S-Shaped  Curve  of  Adoption  from  Rogers,  2003, 

pp.  273 
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Denrell  and  March  (2001)  offer  mathematical  models  of  adoption  and  use 
computer  simulation  to  demonstrate  how  experiential  learning  in  individuals  and 
organizations  can  produce  a  "hot  stove  effect."  A  cat  that  sits  on  a  hot  stove  lid 
once  will  never  sit  one  again  -  but  will  never  sit  down  on  a  cold  stove  lid,  either 
[from  a  parable  by  Mark  Twain],  Denrell  and  March  (2001)  demonstrate  that 
alternatives  to  the  status  quo  (innovations)  often  share  these  characteristics: 

•  initial  performance  (among  novices)  is  lower  than  the  expert 
performance  seen  after  competence  develops  with  the  innovation, 

•  initial  performance  has  higher  variation  than  later  performance  - 
systems  are  more  stable  at  their  final  potential, 

•  variability  and  risk  of  failure  exist  among  alternatives,  and  final 
performance  may  be  substantially  less  than  the  status  quo. 

Denrell  and  March  (2001)  demonstrate  through  simulation  that 
organizations  that  tend  to  persist  in  alternatives  to  reach  a  higher  level  of 
competence  will  therefore  often  persist  in  evaluating  poor  alternatives,  and 
thereby  fail  and  be  weeded  out.  Due  to  high  variability,  risky  alternatives  will  have 
runs  of  poor  returns,  eliminating  individuals  and  organizations  that  take  on  risky 
alternatives.  These  effects  combine  to  produce  a  systematic  bias  in  favor  of 
reliability,  manifested  as  apparently  "risk-averse"  survivors.  The  factors  that 
affect  the  success  of  alternatives  and  of  individuals  who  adopt  them  include: 

•  learning  time  required  to  gain  competence, 

•  variability  in  initial  performance  within  an  alternative  (how  risky  is  a 
particular  alternative?), 

•  variability  in  potential  performance  among  all  alternatives  (how 
much  better  are  the  good  alternatives?), 

•  aspirations  of  desired  performance  (how  much  does  the  decision 
maker  really  want  or  need  to  improve  to  survive?). 

The  factors  that  influence  buyers  deciding  between  products  are  the  same 
as  those  that  influence  adopters  of  other  types  of  innovations.  Much  research 
exists  to  explore  how  successful  marketers  encourage  the  adoption  of  their 
brands  by  consumers. 
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In  contrast  to  analyzing  what  makes  adopters  switch  to  alternatives, 
Carpenter  and  Nakamoto  (1989  and  1994)  examined  how  leading  brands  come 
to  dominate  markets  for  decades.  They  propose  that  pioneering  alternatives 
(first-to-market  innovations)  have  an  advantage  beyond  their  initial  uniquely 
desirable  attributes.  Not  only  can  pioneers  modify  their  product  towards  user 
preferences,  but  they  also  have  the  ability  to  directly  influence  market 
preferences.  Through  experiments  with  actual  products  and  participants, 
Carpenter  and  Nakamoto  (1989)  showed  that  successful  pioneering  products 
could  significantly  influence  user  preferences  in  new  markets.  It  is  then  much 
more  difficult  for  later  competitors  to  differentiate  themselves  from  the  pioneer. 
The  successful  pioneer  shifts  users'  desires  and  preferences  towards  the 
pioneer's  greatest  strengths. 

Carpenter,  Nakamoto,  &  Kent  (1990)  identified  ways  that  later  alternatives 
can  be  made  more  attractive  in  a  market  dominated  by  pioneers.  Shankar, 
Carpenter,  and  Krishnamurthi  (1998)  followed  up  that  research  with  strategies  for 
late  movers  to  reshape  a  market  and  arrest  the  pioneering  brand's  dominance  by 
identifying  a  specific,  superior  position  away  from  the  pioneer. 

User  preferences  for  alternatives  can  be  manipulated  in  other  ways. 
Carpenter,  Glazer,  and  Nakamoto  (1994)  extended  Rogers'  (2003)  principle  of 
relative  advantage  -  that  one  alternative  is  somehow  objectively  better  than 
another  for  the  adopter.  Carpenter  et  al.  (1994)  showed  that  when  alternative 
products  have  no  meaningful  differentiation  between  them  (such  as  shampoo 
and  instant  coffee),  consumers  preferences  can  be  significantly  swayed  by  the 
successful  marketing  of  irrelevant  attributes  that  have  no  impact  on  objective 
product  performance.  Their  experiments  showed  that  consumer  valuation  of 
alternatives  is  not  properly  modeled  by  a  rational  choice  interpretation.  They  also 
showed  that  there  is  significant  interaction  between  the  cost  of  the  alternative 
and  the  perceived  value  of  the  irrelevant  alternative  -  that  is,  a  premium  price 
tends  to  reinforce  the  value  of  an  irrelevant  alternative.  As  a  specific  example, 
Carpenter,  Glazer  and  Nakamoto  (1994)  offered  irrelevant  differences  in  CD 
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players,  pasta  types,  and  down  jackets  to  participants  in  a  controlled  experiment. 
The  strength  of  the  effect,  that  irrelevant  attributes  were  positively  valued  in 
premium  priced  products,  surprised  the  researchers. 

2.  Human-Computer  Interaction 

Functional  system  designs  with  human  interaction  must:  1)  accept  and 
process  user  input,  and  2)  perform  some  functional  task.  This  is  true  for  all 
manner  of  interactive  systems  (mechanical,  electronic,  or  social).  Context-aware 
designs  present  a  different  interface  to  the  user(s)  depending  on  the  immediate 
context  of  use. 

A  context-aware  design  (compared  to  a  context-unaware  interface  for  the 
same  system)  can  improve  user  efficiency  while  decreasing  errors  and  training 
requirements.  Contextual  awareness  is  becoming  increasingly  a  part  of  modern 
user  interfaces,  information  processing,  decision-support,  and  command  and 
control  systems  for  military  applications.  Selker  and  Burleson  (2000)  introduced  a 
framework  for  discussing  computer  designs  that  can  be  extended  to  all 
interactive  system  models: 

•  task  models  (Lieberman  and  Selker,  2000)  are  the  acts  that  the 
user  directly  performs  to  exercise  some  functional  capability  of  the 
system, 

•  user  models  consist  of  task-relevant  background  information  about 
the  user, 

•  system  models  are  the  capabilities  of  the  [computer]  system. 

Norman  (2002)  analyzed  a  number  of  mechanical  systems  in  which  task 

models  are  not  well-defined  for  the  user,  user  models  are  improper  or  ignored, 
and  system  models  are  misleading  or  unavailable  to  the  user. 

Rich  (1999)  argued  that,  “for  many  interactive  computer  systems,  the  user 
community  is  sufficiently  heterogeneous  that  a  single  model  of  a  canonical  user 
is  inadequate.”  She  presents  three  techniques  for  building  user  models: 
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•  identifying  the  vocabulary  and  concepts  employed  by  the  user, 

•  gauging  the  responses  with  which  the  user  seems  satisfied,  and 

•  using  stereotypes  (a  collection  of  user  traits)  to  generate  many 
facts  from  a  few. 

People  are  most  familiar  with  machine  systems  that  produce  direct, 
computational,  and  highly  accurate  results  from  quantitative  input  data.  “Expert,” 
or  knowledge-based,  systems  produce  indirect  conclusions  (often,  qualitative, 
and  in  a  restricted  field)  that  closely  parallels  the  conclusions  drawn  by  a  human 
“expert”  in  that  field.  Experts  (human  or  machine)  are  believed  to  perform  better 
than  “novices”  (non-experts)  in  part  because  they  use  superior  internal  models  of 
the  system  (Jonassen,  2003). 

In  case  studies,  Rich  (1999)  followed  the  development  of  an  expert 
system  that  uses  “stereotypes”  to  individualize  a  user  model.  Cognitive  research 
has  shown  that  a  problem’s  representation  to  the  user  has  a  tremendous  effect 
on  a  subject’s  internal  model  of  the  problem  (Jonassen,  2003;  Norman,  1993). 
Naturalistic  decision  making  (NDM)  theory  asserts  that,  “often  the  biggest 
problem  facing  decision  makers  is  that  they  do  not  know  which  questions  to  ask” 
(Meso,  Troutt,  &  Rudnicka,  2002). 

Context-aware  design  can  help  bring  novices  up  to  expert-level 
performance  by  properly  presenting  the  contextually  relevant  data.  A  user  can 
then  construct  an  internal  model  that  accurately  reflects  the  situation  (in  other 
language,  build  “situational  awareness”).  A  context-aware  design  might  then 
suggest  (or  conversely,  suppress)  possible  courses  of  action,  rather  than 
overwhelm  the  user  (often  termed,  “information  overload”). 

System  models  can  adopt  a  “process-centered”  or  a  “product-centered” 
view  of  the  system  components  (Abecker,  Bernardi,  Hinkelmann,  Kuhn,  &  Sintek, 
2000).  A  context-aware  design  should  be  able  to  gather  data  on  context  not 
explicitly  given  to  it,  such  as:  time,  place,  weather,  user  preferences,  skills, 
knowledge,  and  experience,  and  the  history  of  interaction.  These  definitions  of 
context  can  also  be  extrapolated  to  states  unfamiliar  to  the  system:  not  just, 
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“what  is  likely  in  the  neighborhood  right  now,”  but,  more  generally,  “what  is 
typically  in  a  neighborhood  like  this”  [emphasis  added]  (Lieberman  &  Selker, 
2000). 

Developing  context-aware  designs.  Context-aware  designs,  by 
definition,  respond  to  external  conditions.  They  have  a  degree  of  sophistication 
that  is  unusual  for  machines.  Harter,  Hopper,  Steggles,  Ward,  and  Webster 
(2002)  go  so  far  as  to  describe  a  particular  context-aware  platform  as  “sentient”. 
Colloquially,  the  term  “smart”  is  often  variously  used  to  refer  to  a  system  that: 

•  performs  complex  tasks  with  high  “ease  of  use”,  or 

•  uses  a  knowledge  base  to  produce  conclusions  not  obvious  to  a 
non-expert,  or 

•  attempts  to  predict  or  infer  user  action  (that  is,  at  least  rudimentarily 
context-aware). 

The  first  definition  merely  indicates  the  initial  employment  of  any 
technological  system.  This  level  of  human  factors  engineering  is  well  appreciated 
by  industry  (Vredenburg,  1999  and  2003),  but  has  nothing  to  do  with  contextual 
awareness.  A  marvel  of  complexity  may  produce  incorrect  output,  but  be 
(erroneously)  believed  because  it  bears  a  false  “pedigree  criteria”  for  credibility 
(Vedder,  2004). 

Knowledge-based  systems,  built  on  an  complex  system  model,  must  also 
assess  the  context  of  the  data  presented  to  the  system,  and  the  user’s  state  to 
produce  useful  and  reliable  results.  Suchman  (1987,  ch.  6-7)  described  in  great 
detail  a  case  study  of  an  expert  system  that  has  an  excellent  knowledge  base 
and  internal  system  model,  but  does  not  adequately  collect  information  on  the 
user’s  context.  Because  it  is  context-unaware,  its  comprehensive  knowledge 
base  is  unavailable  to  the  user,  and  the  system  is  a  failure. 

Perception  can  create  reality.  Persuading  users  to  learn  and  use  a 
system’s  capabilities  goes  beyond  mere  functional  effectiveness.  Design  and 
aesthetics  can  affect  a  user’s  perception,  belief  system,  and  even  task  ability 
(Csikszentmihalyi,  1990).  Picard’s  continuing  research  (2004)  in  “affective 
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computing”  has  shown  that  a  critical  part  of  our  ability  to  see  and  perceive  is  not 
logical,  but  emotional.  Her  Affective  Computing  Research  Group  at  the  MIT 
Media  Laboratory  is  studying  the  nature  of  human  emotions,  how  to  sense  and 
model  them  (essential  parts  of  a  complete  user  model),  and  experimenting  with 
“synthesizing  emotions”  in  machines  to  create  affective  interfaces  and 
applications  (Picard,  2004).  “Seductive”  experiences  create  an  emotional  bond 
with  their  audience  (Khaslavsky  &  Shedroff,  1999).  A  seductive  interface 
enhances  the  value  to  users,  and  has  the  goal  of  persuading  users  towards 
some  aim.  Khaslavsky  and  Shedroff  described  three  basic  steps  towards 
seduction:  enticement,  relationship,  fulfillment.  Technologies  now  exist  that  are 
actively  persuasive  in  their  own  right,  such  as  interactive  agents  that  steer  users 
towards  a  company’s  product  (Berdichevsky  &  Neunschwander,  1999). 


Figure  2.  The  IBM  “Out-Of-Box  Experience”  -  a  physically 
context-aware  design  (Selker  &  Burleson,  2000) 


Selker  and  Burleson  (2000)  described  physical,  mechanical  and  electronic 
context-aware  designs.  For  example,  the  setup  instructions  for  an  IBM™  laptop 
computer  are  not  presented  as  a  (context-unaware)  traditional  manual  (Figure  2) 
The  instructions  are  on  strips  of  paper,  in  and  on  various  parts  of  the  computer, 
so  that  the  user  is  presented  with  new  actions  only  when  they  are  useful  (one 
strip  is  on  the  outside,  another  becomes  visible  when  the  machine  lid  is  raised, 
etc.)  Norman  (2002)  called  these  context-based  constraints  “forcing  functions”  - 
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situations  in  which  actions  are  constrained  (the  system  correctly  interprets  the 
user’s  context)  so  that  failure  at  one  stage  prevents  the  next  step  from 
happening.  Norman  (2002)  also  identifies  a  number  of  other  physical  and 
mechanical  designs  that  notably  assist  or  confound  the  user  by  their  degree  of 
contextual  awareness.  Forcing  functions  can  be  misapplied,  and  thereby  prevent 
an  appropriate  means  of  recovery.  This  will  lead  to  users  quickly  discovering 
ways  to  circumvent  the  system  and  fool  the  system’s  user  model,  and  in  doing 
so,  destroy  the  system’s  context  (Suchman,  1987,  ch.  7).  If  users  absorb  such 
actions  into  their  task  model  for  the  system,  both  performance,  and  confidence  in 
the  system,  will  suffer. 


Figure  3. 


The  Traditional  Engineering  and  Science  “Black-Box” 
system  model 


Harter  et  al.  (2002)  describe,  in  great  detail,  the  technical  issues 
concerning  a  specific  hardware  and  software  platform  that  gains  contextual 
awareness  (and  maintains  an  accurate  user  model)  by  sensing  the  user’s 
location  in  a  building.  Abecker  et  al.  (2000)  describe  the  task  models  dealt  with 
by  their  KnowMore  project.  The  KnowMore  project  deals  with  the  problem  of 
“information  overload”.  A  vast  amount  of  knowledge  is  available  through 
information  retrieval  tools  (search  engines),  but  by  contextually  modeling  user’s 
information  needs,  the  system  eliminates  unrelated  documents  and  presents  the 
user  only  with  applicable  knowledge  and  compulsory  rules.  Lieberman  &  Selker 
(2000)  point  out  that  there  is  a  trade-off  between  the  desire  for  abstraction  and 
the  desire  for  context  sensitivity 

Difficulties  in  Designing  Context-Aware  Systems.  Context-based 
designs  break  out  of  the  traditional  engineering  and  science  “black-box”  model  of 
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system  design  shown  in  Figure  3.  Lieberman  and  Selker  (2000)  propose  an 
expanded,  context-aware,  systems  design  model,  which  is  further  modified  in 
Figure  4.  The  traditional  component-wise  black-box  decomposition  is  useful 
because  it  abstracts  a  particular  system  function  from  the  (much  more  complex) 
whole.  Black  boxes  are  often  simpler  to  design,  test,  and  implement.  It  is  exactly 
this  abstraction  that  reduces  context  sensitivity. 


Gacl&aL 

State  of  the  User 
State  of  the  Process 
Physical  Environment 
History  of  Interaction 


Figure  4.  Context  is  everything  but  the  explicit  input  and  output 
(adapted  from  Lieberman  &  Selker,  2000) 


Vedder  (2004)  discussed  the  issues  of  humans  assessing  information 
reliability  (specifically,  information  on  the  internet),  and  in  what  ways  humans  fail 
to  recognize  reliability  or  unreliability.  Highly  accessible  information,  such  as 
results  from  an  internet  search  engine,  is  often  mistakenly  held  to  be  reliable. 
The  fact  that  information  comes  from  the  internet  can  give  it  a  misleading 
“pedigree  of  reliability”  (Vedder,  2004).  Vedder  (2004)  cited  no  research  in  this 
area,  but  his  conceptual  discussion  can  be  extended  to  a  measuring  instrument 
that  displays  more  significant  digits  than  are  actually  accurate,  or  the  display  of  a 
so-called  “smart”  interface.  Humans  are  often  at  the  extremes  in  their  skepticism 
of  machine-displayed  information  -  either  too  credulous,  or  too  dubious.  Good 
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designs  should  incorporate  a  measure  of  reliability  of  their  output,  so  as  not  to 
lead  the  user  into  a  false  context  (of  accurate  data).  Information  density  and  user 
comprehension  can  be  vastly  improved  with  an  appropriate  choice  of  display. 
Tufte  (2001)  illustrated  a  wide  variety  of  ways  (good  and  bad)  to  display 
quantitative,  multi-variate  data. 

Berdichevsky  and  Neunschwander  (1999)  set  out  ethical  principles  of 
persuasive  design,  with  a  “golden  rule”  that,  “actively  persuasive  technology 
should  never  seek  to  persuade  anyone  of  something  they  themselves  would  not 
consent  to  be  persuaded  to.”  People  are  persuaded  (by  other  people)  based  in 
part  on  “liking”  and  “authority”  (Cialdini,  1993).  Within  appropriate  contexts, 
machine  interfaces  can  similarly  be  made  seductive,  affective,  and  authoritative, 
enabling  them  to  persuade  users  towards  (or  against)  certain  actions. 

Displays  that  do  not  take  operator  context  into  account  may  be  ignored  or 
rejected  as  inaccurate.  Besnard,  Greathead,  and  Baxter  (2004)  described  how 
flawed  mental  models  (particularly,  in  a  detailed  study  of  the  British  Airlines  737 
crash  in  1989  in  Kegworth,  U.K.)  can  be  mistakenly  reinforced  by  coincidental 
system  responses. 

“People’s  actions  are  intimately  intertwined  with  artifacts  of  their  work; 
their  team  member’s  roles,  responsibilities,  and  actions;  and  even  their  cultural 
and  historical  setting,”  (Olson  &  Olson,  2003,  p.  495).  Olson  and  Olson  (2003) 
charted  out  how  the  field  of  Human-Computer  Interaction  is  transitioning  from 
early  research  in  individual  cognitive  processes  and  motor  skills,  to  a  knowledge- 
based  understanding  of  the  role  of  context  and  “domain  expertise”  in  user 
interfaces. 

The  costs  of  poor  design.  Hackos  and  Redish  (1998)  point  out  a  number 
of  costs  incurred  by  products  that  are  designed  without  integrated  user  and  task 
analysis.  Cutting  corners  in  the  design  process  can  result  in  lifetime  expenditures 
in: 

•  lost  productivity, 


15 


•  increased  training  and  re-training  time, 

•  maintenance  costs, 

•  user  support  costs, 

•  field  maintenance  costs, 

•  correcting  mistakes, 

•  health  and  safety  problems  due  to  improper  use, 

•  administrative  and  legal  costs  due  to  improper  use, 

•  employee  frustration  leading  to  low  productivity  and  contributing  to 
attrition, 

•  and  lost  sales  or  poor  adoption. 

User  interface  design  conclusions.  Context-aware  designs  seek  to 
improve  user  efficiency  while  decreasing  errors  and  training  requirements. 
Contextual  awareness  and  presentation  are  usually,  but  not  always,  more 
complicated  to  incorporate  into  designs.  Basic  human  factors  considerations 
dictate  that  design  should  follow  the  human’s  point  of  view,  rather  than  the 
convenience  of  technical  design.  Beyond  that  level,  designing  contextual 
awareness  requires  considering  the  system  from  the  human’s  current  point  of 
view.  A  good  context-aware  interface  must  provide  access  to  all  relevant  system 
capabilities,  while  aiding  the  user  in  making  appropriate  choices  and/or  restricting 
the  user’s  ability  to  make  inappropriate  choices.  Poor  attempts  at  context-aware 
interfaces  may  inadvertently  restrict  access  to  important  capabilities,  be  jarringly 
inconsistent,  or  misinterpret  the  user’s  actual  context. 

The  prevailing  literature  addresses  the  advantages  of  incorporating 
contextual  awareness  into  user  interfaces  and  whole-system  designs.  Actual 
examples  of  context-aware  designs  are  primarily  experimental,  or  in  specific  and 
novel  applications.  There  exists  little  in  the  way  of  comparative  studies  of 
improvements  gained  by  applying  context-aware  designs  to  existing,  context- 
unaware  interfaces. 
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Military  information  systems  are  often  used  in  one  or  more  of  the  following 
situations: 

•  Decision  makers  must  make  sense  of  overwhelming  amounts  of 
relevant  and  timely  data. 

•  An  operator  or  watch  officer  must  have  a  formal  “turnover”  period  to 
bring  his  or  her  relief  abreast  of  the  current  situation  (that  is, 
establish  their  context  of  operations). 

•  A  user  has  considerable  freedom  to  decide  how  to  accomplish  a 
specific  task,  and  different  users  are  more  effective  operating  in 
their  own,  personalized  “mode.” 

•  Users  have  greatly  varying  degrees  of  skill  and  training,  but  all  must 
operate  the  same  system  at  a  minimum  proficiency. 

•  A  human,  in  the  position  of  the  system,  would  have  implicit 
knowledge  of  many  environmental  factors  (time,  weather,  historical 
context)  that  would  not  be  explicitly  entered  into  a  system. 

These  situations  all  are  candidates  for  improvement  through  context- 
aware  system  design,  and  context-aware  interfaces.  Context-aware  design 
applied  to  military  individual  equipment  could  be  implemented  in  the  following 
areas. 


Sensible  default  behavior.  The  most  likely  choice  is  presented  to  the 
user.  For  example,  novices  need  not  enter  complicated  networking  configuration 
information  if  the  system  can  determine  some  configuration  that  would  work. 
Advanced  users  might  optimize  their  settings  in  some  other  manner. 

Restricted  options.  For  users  of  various  categories,  based  on  skill  level, 
tactical  role,  or  security  requirements. 

Robustness  to  inputs.  Users  of  tactical  equipment  will  exit  applications 
and  turn  off  the  device  at  any  possible  time.  Components  may  be  disconnected 
at  any  time  during  operation,  and  power  sources  may  be  intermittent.  Devices 
should  be  robust  to  all  such  conditions  without  corrupting  or  deleting  data  or 
physical  damage  to  equipment. 

Informative  error  conditions  and  graceful  recovery.  Error  messages 
should  recommend  an  appropriate  recovery  procedure  (for  example,  a  sensible 
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default  behavior).  A  universal  reset  procedure  should  be  available  to  restore  a 
system  to  normal  operation. 

Users  should  be  drawn  from  typical  current  users  ranging  from  trainees  to 
highly  experienced  experts.  The  demographics  and  psychographics  of  the 
representative  group  should  be  compared  with  those  of  the  larger  population  of 
actual  users,  and  significant  departures  be  noted.  Pre-interviews  with  the 
subjects  should  identify  the  following: 

•  preconceptions  and  biases  about  the  system, 

•  their  existing  understanding  of  the  user,  task,  and  system  models, 
and 

•  previous  experience  with  similar  systems. 

An  aspect  of  contextual  awareness  that  is  not  well  discussed  in  the 
literature  is  the  trade-off  between  better  performance,  using  contextual 
awareness  and  the  benefits  of  standardization  across  users.  This  has  particular 
significance  for  users  of  military  designs.  For  a  number  of  reasons  in  training  and 
wartime,  personnel  changes  often  happen  rapidly  in  military  units.  Replacement 
personnel  may  have  a  wide  range  of  skills  and  experiences.  All  military  systems 
must  be  designed  in  such  a  way  that  users  can  be  substituted  for  one  another 
without  warning,  which  has  implications  for  how  far  context  aware  designs  can 
customize  themselves  to  a  specific  user  model. 

A  primary  purpose  of  this  research  is  not  to  develop  the  “how”  of  system 
design,  but  the  “why”  -  why,  and  by  how  much,  is  a  high  technology  solution 
better  than  a  traditional  system?  In  what  areas  does  high  technology  strongly 
benefit  the  overall  system,  and  in  what  areas  are  diminishing  returns  seen  from 
applying  high  technology  solutions?  The  answers  to  these  questions  will 
stimulate  further  research  with  at  least  two  goals:  1)  to  improve  specific  existing 
systems,  and  2)  to  encourage  system  designers  to  incorporate  usability  analysis 
in  future  system  designs  from  their  inception.  As  with  other  human  factors 
improvements,  it  is  far  easier  to  create  a  system  with  the  proper  perspective  in 
mind  from  the  beginning,  than  to  retrofit  poor  designs  later. 
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3.  Usability  Testing 

In  the  standards  document  ISO  9241-11,  the  International 
Standards  Organization  (ISO)  defines  usability  and  three  related 
terms: 

Usability:  The  effectiveness,  efficiency,  and  satisfaction  with  which 
specified  users  can  achieve  specified  goals  in  particular 
environments. 

Effectiveness:  the  extent  to  which  a  goal,  or  task,  is  achieved. 

Efficiency:  the  amount  of  effort  required  to  accomplish  the  goal. 

Satisfaction:  the  level  of  comfort  that  the  user  feels  when  using  a 
product  and  how  acceptable  the  product  is  to  users  as  a  vehicle  for 
achieving  their  goals. 

In  his  brief  and  very  readable  An  Introduction  to  Usability  (1998),  Jordan 
identified  and  discusses  the  user  characteristics  that  have  the  most  direct  effects 
on  usability:  experience,  domain  knowledge,  and  cultural  background.  Jordan 
(1998)  also  included  disability,  age  and  gender  in  his  list,  but  within  the  narrower 
context  of  military  usability  studies,  these  characteristics  are  less  likely  to  be 
relevant. 

There  are  various  taxonomies  of  errors  (see  Wiegmann  &  Shappell,  2003, 
Reason,  1990,  Dekker,  2002,  and  Norman,  2002).  The  errors  commonly 
measured  in  usability  testing  include  slips  and  mistakes.  A  slip  occurs  when  the 
user  knows  how  to  perform  a  task,  but  unintentionally  does  something  wrong 
during  the  task.  A  mistake  occurs  when  the  user  has  an  erroneous  model  of  how 
the  product  works  (Reason,  1990).  Slips  and  mistakes  may  indicate  different 
reasons  for  the  commission  of  an  error,  but  may  be  difficult  to  distinguish  in  some 
contexts. 

As  illustrated  in  Figure  5,  Jordan  (1998)  defines  five  components  of 
usability  in  a  manner  pertinent  to  the  analysis  of  military  systems  (boldface  and 
italics  added): 
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Guessability.  The  effectiveness,  efficiency,  and  satisfaction  with  which 
specified  users  can  complete  specified  tasks  with  a  particular  product  for  the  first 
time. 

Learnability.  The  effectiveness,  efficiency,  and  satisfaction  with  which 
specified  users  can  achieve  a  competent  level  of  performance  on  specified  tasks 
with  a  product,  having  already  completed  those  tasks  once  previously. 

Experienced  User  Performance  (EUP).  The  effectiveness,  efficiency, 
and  satisfaction  with  which  specified  experienced  users  can  achieve  specified 
tasks  with  a  particular  product. 

System  Potential.  The  optimum  level  of  effectiveness,  efficiency,  and 
satisfaction  with  which  it  would  be  possible  to  complete  specified  tasks  with  a 
product. 


Re-usability.  The  effectiveness,  efficiency,  and  satisfaction  with  which 
specified  users  can  complete  specified  tasks  with  a  particular  product  after  a 
comparatively  long  period  away  from  those  tasks. 


EXPERIENCE  WITH  PRODUCT 

Figure  5.  The  components  of  usability,  and  an  idealized 
learning  curve  (Jordan,  1998). 

Why  not  simply  ask  users  or  subject  matter  experts  (SMEs)  what  they 
want?  Hackos  and  Redish  (1998)  point  out  that  users  themselves  do  not  know 
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how  to  articulate  what  they  do,  especially  if  they  are  very  familiar  with  the  tasks 
the  perform.  Users  will  tend  to  emphasize  difficult,  boring,  or  exciting  tasks. 


D.  OBJECTIVES 

The  objective  of  this  research  effort  is  to  design  and  establish  a  laboratory 
and  research  method  specifically  focused  on  scientifically  evaluating  the  human 
factors  considerations  of  military  individual  equipment  under  both  laboratory  and 
field  conditions.  This  laboratory  will  represent  the  first  of  its  kind  for  academic  and 
military  human  factors  research.  The  specific  results  of  this  effort  include: 

•  a  complete,  well-tested,  and  professional  usability  laboratory 

designed  specifically  for  evaluating  military  individual  equipment, 

•  a  rugged,  embarkable,  fully  self-contained  portable  usability 

laboratory  for  field  research, 

•  a  codified  manual  for  using  the  two  main  configurations  (stationary 
and  portable)  of  the  usability  laboratory,  written  for  the  beginning 
usability  researcher, 

•  a  tested  set  of  procedures  for  applying  sound  human  factors 
principles  to  the  specific  problem  of  usability  testing  of  military 
individual  equipment, 

•  a  proof-of-concept  practical  application  of  the  laboratory  and 

procedures  to  a  specific  problem,  namely  the  usability  testing  of 
RPDAs  designed  for  ground  combat  personnel. 

E.  SCOPE,  LIMITATIONS,  AND  ASSUMPTIONS 

This  research  effort  was  limited  primarily  by  time  and  the  availability  of 
appropriate  participants.  In  order  to  both  design  and  fully  exercise  the  capabilities 
of  the  laboratory,  this  thesis  was  intended  to  be  limited  as  follows. 

Review  the  extant  literature  pertinent  to  the  human  factors  of  RPDAs. 
Identify  military  RPDAs  currently  under  development. 

Discover  the  requirements  for,  design,  construct,  and  create  training  for  a 
usability  laboratory  suitable  for  testing  and  evaluating  handheld  electronic 
devices. 
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Define  a  methodology  for  evaluating  the  human  factors  considerations  of 
military  PDAs,  including  the  software  and  hardware  of  behavioral  observation 
systems,  data  collection  procedures,  and  training. 
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II.  DESIGN  AND  IMPLEMENTATION  OF 
THE  OBSERVATION  LABORATORY 


A.  BACKGROUND 

The  bulk  of  the  time  in  this  effort  was  spent  in  designing,  constructing, 
learning  about,  and  experimenting  with  the  equipment  used  for  the  Applied 
Warfighter  Ergonomics  (AWE)  Observation  Laboratory.  The  technical  details  of 
the  installation  and  use  of  the  lab  are  given  in  step-by-step  detail  in  the  Applied 
Warfighter  Ergonomics  Observation  Laboratory  Basic  User  Guide  (Appendix  B, 
also  published  as  a  stand-alone  document). 

During  the  pilot  experiments  leading  up  to  the  case  study  and  during  the 
case  study  itself,  students  were  brought  in  to  use  the  equipment  in  the  AWE 
Observation  Labi.  These  students  exercised  the  various  capabilities  of  the  lab, 
including  capturing  and  editing  video,  creating  DVDs,  creating  configurations  in 
The  Observer  software,  and  coding  behavior  in  The  Observer  software.  They 
referred  only  to  the  Basic  User  Guide  created  during  this  project  (found  in 
Appendix  B),  with  occasional  coaching  where  necessary.  Feedback  during  this 
process  refined  the  Basic  User  Guide  to  the  point  where  the  last  users  required 
no  coaching  beyond  the  tutorial  in  the  Basic  User  Guide  to  fully  perform  a  basic 
usability  study. 

The  Naval  Postgraduate  School  Applied  Warfighter  Ergonomics 
Observation  Laboratory  (NPS  AWE  Observation  Lab)  was  designed  to  be  the 
first  observation  laboratory  designed  explicitly  to  investigate  military  individual 
equipment.  Traditional  observation  laboratories  exist  in  the  civilian  community 
that  are  used  variously  to  study  computer  interfaces,  websites,  and  small 
physical  devices  (Noldus,  2005).  There  already  exist  many  (often  conflicting) 
human  interface  guidelines  for  software,  as  well  as  detailed  ergonomic  studies  of 


1  Students  who  used  the  AWE  Observation  Laboratory  during  its  initial  testing  included:  NPS 
Human  Systems  Integration  students  Petra  Alfred,  Tim  McKnelly,  Derek  Read,  and  Matt 
Simpson,  and  UCSD  undergraduate  cognitive  science  student  Katie  Buettner. 
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highly  sophisticated,  expensive  military  equipment  such  as  aircraft  cockpits  and 
nuclear  submarine  control  stations.  Currently,  no  usability  lab  exists  which  is 
dedicated  to  the  study  of  the  equipment  carried  by  a  single  soldier  or  Marine,  and 
the  behavior  and  characteristics  of  that  ground  combat  Warfighter. 

The  AWE  Observation  Laboratory  was  conceived  with  this  mission 
statement: 

To  design  and  establish  a  laboratory  and  a  methodology  for 
scientifically  evaluating  the  human  factors  considerations  of  combat 
equipment  used  by  military  personnel  under  both  laboratory 
controlled  and  realistic  field  conditions. 

The  state  of  the  art  in  behavioral  observation  hardware  and  software  was 
sought,  in  conjunction  with  the  appropriate  military  equipment  to  focus  the  lab  on 
military  individual  equipment.  The  following  is  a  general  list  of  the  functions  that 
were  required  to  establish  this  laboratory  as  uniquely  serving  the  ground 
combatant: 

•  robotic  digital  cameras  to  record  behavior  from  multiple  angles, 

•  video  mixing  and  editing  hardware  and  software, 

•  standard  issue  military  individual  equipment,  such  as  helmets,  body 
armor,  packs,  and  training  rifles, 

•  large  screen  plasma  display  for  presenting  tactical  scenarios  to 
participants, 

•  wired  and  wireless  network  access  to  the  Naval  Postgraduate 
School  research  network, 

•  GPS  rebroadcasting  equipment  allowing  the  reception  of  real  GPS 
signal  inside  the  laboratory  (where  GPS  signals  are  usually 
unavailable), 

•  wired  and  wireless  network  access  to  the  NPS  “Gigalab”  Tactical 
Network  Topology  experimental  research  network  so  that  the  latest 
networked  computing  devices  developed  at  NPS  could  operate 
within  the  lab. 

Establishing  these  capabilities  required  close  coordination  with  the 
following  individuals  and  agencies. 
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•  Dr.  Dave  Netzer’s  Tactical  Network  Topology  (TNT)  research 
program,  including  a  number  of  students  involved  in  the  quarterly 
TNT  experiments, 

•  Dr.  Alex  Bordetsky’s  Gigalab  research  group, 

•  Captain  Francisco  Caceres,  student  in  the  NPS  Information 
Technology  Management  curriculum,  for  his  work  with  RPDAs, 

•  the  Marine  Corps  Tactical  Systems  Support  Activity  in  San  Diego, 
California, 

•  Noldus  Information  Technology  of  Leesburg,  Virginia,  developer  of 
The  Observer  software  and  related  observation  hardware, 

•  NPS  Public  Works  department  for  the  installation  of  a  two-way 
mirror,  a  GPS  antenna  on  the  roof  of  Glasgow,  telephone  systems, 
and  other  small  construction  projects, 

•  NPS  Information  and  Technology  Assistance  Center  (ITACS)  for 
the  assignment  of  static  IP  addresses  to  various  equipment  and 
network  connectivity  to  the  NPS  domain  for  the  computer  systems, 

•  Army  and  Marine  Corps  staff  at  the  Defense  Language  Institute 
(DLI)  at  the  Presidio  of  Monterey  (POM),  for  access  to  DLI  students 
as  experimental  participants. 

Detailed  point  of  contact  information  for  the  individuals  and  agencies  listed 

above  can  be  found  in  Appendix  D. 

B.  TECHNICAL  SPECIFICATIONS 

A  detailed  list  of  all  the  equipment,  by  manufacturer  and  model  number, 
and  complete  wiring  diagrams  can  be  found  in  the  Applied  Warfighter 
Ergonomics  Observation  Laboratory  Basic  User  Guide  (Appendix  B).  The  user 
guide  also  contains  a  complete  step-by-step  tutorial  that  would  take  an 
inexperienced  user  through  the  laboratory  hardware  and  software.  An 
accompanying  DVD  shows  the  equipment  and  procedures  for  using  the 
laboratory. 


C.  INSTALLATION 

Initial  installation  of  the  Observer  computer  system  was  performed  in  a 
single  day  by  Koichi  Takagi  of  NPS,  and  Bart  von  Roekel  and  John  McGraw  of 
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Noldus  Information  Technology.  In  addition  to  the  observation  equipment 
purchased  from  Noldus,  the  basic  requirements  for  the  observation  lab  included: 

•  two  adjacent  rooms  (the  “participant”  and  “observer”  room) 
measuring  at  least  10’  x  10’.  A  much  larger  room  is  preferred  for  the 
participant  room.  Each  room  should  be  able  to  be  fully  isolated  from 
other  areas,  and  have  comfortable  lighting  and  temperature 
controls.  Each  room  must  have  both  electrical  and  computer 
network  outlets.  The  observer  room  may  require  as  many  as  twenty 
surge-protected  electrical  outlets  for  the  observation  equipment  and 
supporting  various  equipment  for  testing  (battery  chargers,  radios, 
etc.) 

•  access  between  the  rooms  (for  example,  through  a  suspended 
ceiling)  for  a  considerable  amount  of  cabling, 

•  wall-mounted  remote  cameras  and  their  associated  cabling, 

•  installation  of  a  piece  of  two-way  mirrored  glass  between  the 
rooms.  This  step  required  extensive  coordination  with  local  facilities 
maintenance.  The  glass  was  installed  with  the  lower  edge 
approximately  36  inches  above  the  floor,  so  that  a  seated  observer 
would  have  full  view  of  the  participant  over  his  or  her  desk.  The 
observer  room  must  be  darkened  to  fully  prevent  the  participant 
from  seeing  the  observer  through  the  glass.  Blinds  were  installed  in 
front  of  the  two-way  mirror,  and  shelves  installed  near  the  cameras, 
to  reduce  the  participant’s  self-consciousness  and  awareness  of 
the  observer. 

The  observation  equipment  requires  ample  shelf  and  desk  space  for 
computers,  multiple  video  monitors,  the  camera  controller  and  video  mixer,  note¬ 
taking  materials,  and  other  equipment.  Additional  adjustments  and  improvements 
to  the  laboratory  space  required  approximately  forty  additional  hours  spread  over 
many  months. 
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III.  APPROACH  FOR  USABILITY  TESTING 


A.  OVERVIEW 

This  thesis  is  intended  to  dramatically  reduce  the  demands  of  performing 
effective  usability  research  on  military  individual  equipment  by  providing:  1) 
equipment  customized  to  the  task,  2)  a  validated,  step-by-step  procedure  for 
performing  all  phases  of  the  research  from  data  collection  to  analysis,  and  3)  a 
template  (the  case  study  in  part  IV)  for  presenting  such  a  study. 

Table  1  lists  the  steps  in  performing  a  usability  analysis  of  military 
individual  equipment  using  the  Applied  Warfighter  Ergonomics  Center 
Observation  Laboratory.  The  rest  of  this  section  will  describe  each  step  in  further 
detail. 


Table  1.  Steps  in  Performing  a  Usability  Analysis 

1 .  Concept 

2.  Definition  of  need 

3.  Design  of  lab  study 

4.  User  identification 

5.  Focus  groups/interviews 

6.  Pre-surveys 

7.  Lab  testing  -  obtain  quantitative  performance  metrics 

8.  Post-operation  survey  (operation  in  lab)  -  qualitative  user 

impressions 

9.  Analysis 

10.  Design  of  field  study 

11.  Field  testing  -  validation  of  laboratory  results  through 
quantitative  metrics 

12.  Post-operation  survey  (operation  in  the  field)  -  qualitative 
user  impressions 

13.  Report 
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B.  SELECTING  PARTICIPANTS 

Participants  in  any  usability  study  should  represent  a  sample  of  the 
intended  user  population.  An  irrelevant  test  population  is  a  threat  to  external 
validity,  to  generalizability,  and  to  credibility  with  the  military  consumers  of  the 
final  report.  Assumptions  about  true  user  populations  and  test  user  populations 
should  be  determined,  clearly  stated,  and  tested  where  possible  to  maintain  the 
validity  of  the  study. 

A  qualitative  profile  of  the  typical  user  population  can  be  developed 
through  focus  groups,  and  interviews  with  users,  subject  matter  experts,  and  their 
leadership,  and  surveys.  Test  scores  and  quantitative  ergonomic  data,  such  as 
physical  measurements  and  capabilities,  can  also  be  used  to  define  a  working 
“stereotype”  of  the  user  population  (Rich,  1999). 

C.  CONDUCTING  INTERVIEWS  AND  FOCUS  GROUPS 

Interviews  and  focus  groups  are  a  method  used  to  gain  insight  into  issues 
concerning  newly  designed  equipment.  Participants  may  include:  sponsors, 
users,  developers  and  subject  matter  experts.  The  interviews  and  focus  group 
discussions  are  used  to  determine  things  such  as:  user  needs,  assumptions  and 
biases,  past  experiences  of  the  stakeholders,  and  the  conditions  under  which  the 
devices  may  used.  Focus  groups,  interviews,  and  casual  conversations  with  the 
stakeholders  may  reveal  issues  that  would  not  surface  elsewhere.  Cozby  (2004) 
notes  that  the  group  interaction  in  a  focus  group  can  stimulate  a  variety  of 
responses  that  might  not  surface  in  surveys,  interviews,  or  testing. 

D.  ADMINISTERING  SURVEYS 

Bordens  and  Abbott  (2002)  and  Cozby  (2004)  describe  many  of  the  issues 

around  conducting  effective  survey  research.  In  the  surveys  designed  for  this 

research,  both  open-  and  closed-ended  questions  were  used  to  get  as  much 

information  as  possible  from  concise  (3-5  page)  forms.  The  limited  availability  of 

relevant  test  participants  makes  it  important  that  the  survey  questions  be  clear 

and  unambiguous,  and  not  be  overly  time-consuming.  As  suggested  by  Bordens 
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and  Abbott  (2002),  demographic  items  are  not  presented  first  on  the 
questionnaire,  and  related  items  are  presented  together.  For  consistency  and 
ease  of  scoring,  a  five-point  Likert  scale  ("very  dissatisfied"  to  "very  satisfied")  is 
often  used  to  gauge  qualitative  responses. 

In  most  situations,  criteria  for  grouping  the  individuals  may  not  be 
apparent  until  after  the  survey  data  is  collected,  and  the  researcher  must  take 
care  not  to  prematurely  group  responses  into  categories.  A  question  such  as, 
“How  many  hours  per  week  do  you  play  video  games,”  is  better  answered  with  a 
continuous  response  (e.g.,  “enter  number  of  hours”)  than  with  pre-assigned 
categories  (e.g.,  “0-1  hours,  1-3  hours,  more  than  3  hours”)  to  avoid  ceiling  and 
floor  effects.  After  the  data  are  collected,  the  responses  can  be  grouped  into  low, 
medium,  and  high  levels,  if  desired.  Note  that  some  types  of  statistical  analysis 
are  more  appropriate  with  continuous  values  (the  actual  response  in  hours)  than 
with  categorical  (low,  medium,  high).  However,  there  may  be  reason  to  suspect 
that  such  survey  data,  based  on  estimates  and  individual  recollection,  is  not 
really  providing  the  numerical  accuracy  implied  in  the  response.  In  that  case, 
grouping  responses  into  categorical  levels  may  be  more  accurate. 
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Table  2.  Checklist  for  successful  survey  administration 

1 .  Initial  concept  of  survey  -  goals,  participants,  timeframe 

2.  Institutional  Review  Board  (IRB)  approval) 

3.  Review  survey  internally  (peers  and  faculty) 

4.  Administer  pilot  survey  -  edit  and  refine  survey 

5.  Administer  actual  survey 

6.  Provide  plenty  of  copies  of  all  paperwork 

7.  Provide  pencils  or  pens,  and  clipboards  or  a  writing  surface  if 
necessary 

8.  Introduction  and  instruction  letters  to  participants 

9.  Allow  adequate  time  to  complete  survey 

10.  Prepare  folders  and  portable  storage  (filing  box,  etc.)  for  copies 

of  blank  forms  and  surveys,  and  for  completed  forms  and 
surveys  (see  the  example  in  Table  8). _ 


1.  Pre-operation  Surveys 

Users  should  complete  a  survey  before  using  the  equipment  to  assess 
their  demographic  background,  relevant  experience  and  skills,  and  potential 
biases.  The  survey  responses  can  be  aggregated  with  the  goal  of  identifying 
similar  groups  of  users,  such  as  technical  novices  and  experts,  or  early  adopters 
and  hold-out  traditionalists.  Analysis  of  the  responses  may  also  uncover  specific 
independent  variables  (IVs)  that  are  indicators  or  influences  on  final 
performance. 

2.  Post-operation  Surveys 

Users  should  complete  a  survey  after  using  the  equipment  to  obtain 
qualitative  feedback  from  the  user  on  the  equipment.  The  results  of  this  survey 
can  be  used  in  conjunction  with  the  usability  testing  to  determine  how  well  users' 
impressions  of  their  performance  and  satisfaction  with  the  equipment 
corresponds  to  their  measured  performance.  As  with  the  pre-operation  survey, 
the  responses  can  be  used  to  identify  categories  of  users,  by  their  self-reported 
success  and  satisfaction  measures.  Of  special  interest  would  be  areas  where 
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users  self-reports  (e.g.,  survey  or  verbal  feedback)  disagreed  with  their 
measured  performance.  For  example,  a  user  expressing  a  high  level  of 
dissatisfaction  with  a  device  might  exhibit  a  higher  level  of  performance  on  that 
device  than  on  a  device  with  which  she  was  more  satisfied. 

E.  CONDUCTING  A  FUNCTIONAL  TASK  ANALYSIS  AND  COGNITIVE 

TASK  ANALYSIS 

Tasks  performed  during  testing  should  be  as  closely  related  as  possible  to 
the  real-world  tasks  of  the  user.  These  tasks  can  come  from  a  formal  operational 
requirements  document  (ORD),  from  less  formal  statements  gathered  from  focus 
groups  and  interviews,  or  empirically,  from  a  task  analysis  conducted  as  a  part  of 
the  research  effort.  A  functional  task  analysis  may  also  be  used  to  validate  and 
refine  in  detail  the  tasks  defined  in  the  ORD  or  from  other  sources. 

The  functional  task  analysis  of  how  the  equipment  tested  is  actually  used 
will  identify  the  tasks  that  are  most  relevant  to  the  equipment’s  operation,  and  the 
tasks  that  are  practical  to  test  quantitatively.  The  task  analysis  may  also  clarify 
the  allocation  of  functions  within  a  larger  context.  Some  tasks  are  best  performed 
by  a  human,  some  are  better  performed  on  a  device,:  these  divisions  may  be 
situationally  dependent. 

Hackos  and  Redish  (1998)  and  Kirwan  and  Ainsworth  (1992)  detail  many 
of  the  various  techniques  for  conducting  task  analyses,  including  verbal 
protocols,  observations,  and  task  decomposition.  Rubin  (1994)  and  Cozby 
(22004)  present  concise  and  effective  recommendations  for  conducting  surveys, 
usability  testing,  and  experimental  design.  Their  texts  were  used  extensively  in  all 
phases  of  this  research. 

F.  CONDUCTING  AN  OBSERVATIONAL  USABILITY  ANALYSIS 

The  process  of  turning  human  behavior  and  observed  events  into  a 

quantitative  data  set  is  called  coding  or  scoring.  Coding  can  involve  numerical 

data  such  as  recording  times  and  counting  behaviors.  Behavior  can  also  be 

coded  in  categorical  terms,  such  as  types  of  behaviors,  or  be  more  subjective, 
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such  as  gauging  emotions  or  intent.  The  goal  of  such  an  analysis  is  a  written 
record  that  accurately  characterizes  the  experimental  session. 

A  usability  test  should  define  tasks  to  be  evaluated,  the  precise  conditions 
under  which  they  will  be  performed,  and  specific  standards  that  quantify  success 
or  otherwise  measure  performance.  Usability  data  can  be  collected  in  the 
laboratory  or  in  the  field.  Table  3  lists  the  important  characteristics  of  laboratory 
and  field  research  in  the  context  of  usability  research. 


Table  3.  Characteristics  of  Laboratory  and  Field  Research 


Laboratory  research 

Field  research 

Controlled  conditions 

Uncontrolled  conditions 

Repeatable  scenarios 

Unrepeatable  scenarios 

Artificial 

Realistic 

Can  collect  a  large  amount  of  highly 

Reveals  data  not  seen  in  lab 

detailed  data 

Expensive  in  time  and  other  resources 
More  errors  in  data  collection 

Many  confounds 

Allows  statistically  rigorous  conclusions 

Validation  of  laboratory  results 

1.  Tasks 

Tasks  to  be  evaluated  are  legitimate  when  they  are  defined  in  a 
supportable  manner  and  validated  through  an  independent  functional  task 
analysis.  To  the  greatest  degree  possible,  tasks  should  reflect  real-world 
employment  of  the  equipment  tested  within  the  artificial  conditions  of  a 
laboratory,  safety  and  practical  restrictions  in  a  field  exercise,  and  the  limits  of 
direct  measurability.  When  performing  a  comparative  analysis,  tasks  should  be 
limited  to  those  that  can  be  performed  similarly  in  each  testing  condition,  whether 
those  be  different  devices  or  different  groups  of  users. 

Within  the  Observer  software,  each  task  is  defined  as  a  separate 
behavioral  class  containing  two  state  behaviors:  ON  task  and  OFF  task.  When 
defined  as  state  behaviors,  the  software  records  the  period  that  the  task  is 
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performed  as  a  time  interval  (rather  than  just  as  two  discrete  points  in  time), 
which  is  useful  for  later  analysis. 

2.  Conditions 

Precisely  defined,  rigorously  controlled  experimental  conditions  ensure 
that  the  research  has  sufficient  control  to  maintain  internal  validity,  that  the 
experiments  can  be  replicated,  and  the  data  can  be  appropriately  analyzed. 
Controlling  situations  involving  human  behavior  can  be  more  complicated  that  it 
seems,  even  within  the  structure  of  a  laboratory  experiment.  Conditions  should 
be  recorded  to  the  level  of  detail  that  they  can  be  replicated  by  future  researchers 
with  no  further  details  provided.  At  a  minimum,  researchers  must  write  a  detailed, 
scripted  scenario,  that  includes: 

•  a  timeline, 

•  a  description  of  what  each  researcher  and  participant  will  be  doing 
at  each  point  in  the  scenario, 

•  detailed  notes  on  test  equipment  settings, 

•  detailed  notes  on  environmental  conditions,  such  as  lighting,  and 
time  of  day, 

•  indications  of  what  will  trigger  each  event  or  behavior  of  interest, 
that  is,  how  does  the  participant  know  to  start  a  task? 

•  what  equipment  will  be  worn  or  used,  such  as  helmet,  rifle,  or 
handheld  device. 

The  conditions  under  which  the  experiment  is  performed  should  be 
recorded  as  independent  variables  (IVs)  for  the  session  and  possibly  used  in 
later  analysis. 

3.  Standards:  Measures  of  Effectiveness  and  Measures  of 
Performance 

Tasks  and  behaviors  within  a  usability  test  should  be  evaluated  using 
unambiguous,  precisely  defined  standards  coded  by  independent  researchers. 
Common  usability  measures  are:  speed  and  accuracy  (task  time),  and  accuracy 
(#  of  errors  by  type).  Other  human  factors  metrics,  which  are  more  difficult  to 
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measure,  include:  initial  training  time  required,  skill  retention  over  time,  and  a 
device’s  physical  accessibility  or  ergonomics. 


Measures  of  effectiveness  (MOEs),  measures  of  performance  (MOPs), 
and  data  requirements  have  precise  definitions  within  military  Test  and 
Evaluation  (Hoivik,  2005): 

MOE.  A  measure  which  expresses  the  extent  to  which  a  combat 
system  accomplishes  or  supports  a  mission  or  task.  MOEs  are 
designed  to  address  an  issue,  and  should  be  relevant,  complete, 
precisely  defined,  and  quantifiably  measured.  MOEs  should  be 
directly  traceable  to  a  requirements  document. 

MOP.  A  quantitative  or  qualitative  measure  of  a  system’s 
capabilities  or  specific  performance  function.  MOPs  are  often 
expressed  as  distances,  times,  rates,  or  ratios 

Data  Requirement.  A  number  or  other  measurement  which  can  be 
collected  at  a  single  location,  does  not  depend  upon  recall  or 
judgment  by  the  data  collector,  and  can  be  derived  directly  from  the 
MOPs. 

4.  Overview  of  a  Usability  Test  in  the  NPS  AWE  Observation 

Laboratory 

To  use  the  AWE  Center  Observation  Lab  effectively,  researchers  should 
prepare  the  following: 

a.  a  well-formed  research  question  to  investigate.  This  question 
may  emerge  from  a  formal  requirements  document,  or  from  a  focus  group  or 
brainstorming  session, 

b.  a  set  of  the  proposed  independent  variables  (IVs)  that  will  be 
controlled  in  each  observation  session.  This  list  may  include  information  about 
the  participant  (name,  rank,  gender,  etc.)  and  information  about  the  scenario  or 
environment  ("wearing  equipment  load  A",  "night  conditions",  etc.), 

c.  a  list  of  tasks  to  be  given  to  the  participant  (e.g.,  "send  a 

message"), 
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d.  a  detailed  list  of  "behaviors"  (or  "events")  that  will  be  observed 
and  recorded.  Virtually  all  possible  behaviors  of  interest  should  be  defined 
beforehand  ("start  task",  "press  button  1","user  commits  error"...,  "complete 
task").  While  not  all  events  will  occur  in  each  scenario,  some  events  may  occur 
multiple  times. 

e.  a  detailed,  scripted  scenario,  as  previously  described, 

f.  any  required  paperwork  such  as  IRBs,  consent  forms, 
background  questionnaires,  etc. 

Researchers  will  produce  video  files  of  the  observation  session  for  each 
participant,  and  use  Noldus  Observer  to  score  the  behaviors  observed.  The  data 
can  be  exported  in  standard  formats  (such  as  Microsoft  Excel  or  comma- 
separated  values),  and  analyzed  with  Noldus  Observer’s  built  in  statistics 
package  or  external  statistics  programs  of  choice. 

Multiple  coders  or  researchers  can  score  the  same  sessions  from  video 
files,  and  then  conduct  a  reliability  analysis  between  the  coders  (inter-coder 
reliability).  The  same  coder  can  be  compared  against  previous  observations  to 
measure  intra-coder  reliability. 

The  main  phases  of  using  the  observation  laboratory  are  to  set  up  the 
experiment,  record  the  session  to  video,  score  the  behaviors  observed  during  the 
session,  and  analyze  the  data.  Table  4  provides  a  checklist  for  successfully 
performing  such  an  observation  session. 

Practice  runs  before  actually  attempting  to  collect  data  should  be 
considered  absolutely  necessary. 

Conducting  experiments  with  real  human  participants  and  erratic 
equipment  may  introduce  last-minute  changes  to  the  experimental  design.  When 
at  all  possible,  contingencies  (such  as  malfunctioning  equipment  and  missing 
participants)  should  be  planned  for  in  advance,  rather  than  reacted  to  under 
pressure. 
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Table  4.  Checklist  for  setting  up  a  usability  analysis  in  the  NPS  AWE 

Observation  Laboratory 

1.  General 

1.1.  Reserve  the  AWE  Observation  Lab  well  in  advance. 

1.2.  Confirm  the  date  and  time  with  all  participants  and  others  involved. 

1.3.  Open  HSIL  main  room  (GL-221,  phone  656-3138),  participant  (GL-221a), 
and  observer  (GL-216,  phone  656-3134)  rooms. 

1.3.1.  Check  lighting  and  inspect  area  for  cleanliness  and  safety  issues. 

1.3.2.  Secure  any  equipment,  papers,  etc.  that  could  interfere  with  your 
session  or  confuse  participants. 

1.3.3.  Put  up  “RECORDING  IN  PROGRESS”  signs  and  inform  others  in 
the  area  of  your  work. 

1 .4.  Review  the  script  or  run  profile(s)  for  the  observation(s)  to  be  made. 

1.5.  Brief  participant(s)  on  cameras,  microphones  and  general  lab  procedures. 

1 .6.  Ensure  consent  forms  are  properly  filled  out  and  on  file. 

2.  Set  up  participant  room  (GL-221  a) 

2.1.  Arrange  furniture  in  accordance  with  script.  Pay  attention  to  camera 
locations  and  try  to  keep  the  participants’  normal  movements  from 
blocking  the  cameras.  Tape  marks  on  the  floor  to  indicate  where 
participants  should  stand. 

2.2.  Have  an  assistant  stand  on  the  marked  spot  while  you  preset  the  camera 
positions. 

2.3.  Provide  paper,  pencils,  and  other  props  (as  required). 

2.4.  If  used,  turn  on  computer  (power  strip,  CPU,  display). 

2.5.  Participant  workstation  operations  check  (as  applicable). 

3.  Set  up  observer  room  (GL-216) 

3.1.  Have  script,  notepad,  and  pens  available  for  taking  notes  during  the 
session. 

3.2.  Turn  on  power  to  three  surge  protectors,  power  button  on  computer. 

3.3.  Visually  check  that  power  is  on  to:  surge  protectors,  computer,  computer 
speakers,  printer,  event  logging  keyboard,  two  amps,  VCR,  two  monitors, 
camera  controller,  and  video  mixer. 

3.4.  Log  on  to  the  Observer  workstation  as  Noldus  Observer  (no  password 
required). 
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5.  Challenges  in  Coordination 

A  usability  experiment,  by  nature,  involves  human  test  participants, 
unfamiliar  tasks,  and  untested  equipment.  The  researcher  should  always  prepare 
for  unforeseen  and  time-consuming  delays  in  bringing  these  pieces  together. 

When  conducting  any  study  involving  human  participants,  researchers 
should  do  their  utmost  to  incorporate  flexibility  around  the  participants  in  all 
phases  of  the  study.  The  experiment  should  impose  minimal  demands  on  test 
participants.  Their  interest  in  participation  (and  suitability  as  participants)  wanes 
quickly  if  they  must  wait  excessively  due  to  equipment  malfunctions  or 
scheduling  conflicts.  Participants  may  be  unavailable  during  the  planned  testing 
period. 

While  the  usability  researcher  may  be  expert  in  running  a  usability  test,  the 
specific  tasks  used  in  the  experiment  may  be  wholly  unfamiliar.  Pilot  studies  are 
key  in  identifying  the  missing  pieces  of  task  descriptions  that  were  left  out  during 
subject  matter  expert  interviews.  The  usability  researcher  is  not  likely  to  become 
an  expert  in  an  unfamiliar  domain  during  the  experiment,  but  must  have  at  hand 
precise  and  replicable  definitions  of  all  tasks,  conditions,  and  standards.  These 
should  be  reviewed  extensively  with  the  research  sponsor  or  other  domain  expert 
to  ensure  that  the  experiment  will  actually  observe  and  test  the  tasks  and 
behaviors  of  interest. 

The  researcher  should  not  strive  to  become  a  technical  expert  on  the 
product  or  device  being  evaluated.  Many  products  tested  will  be  unstable 
prototypes  or  highly  technical  in  nature.  A  domain  expert  on  the  device  should  be 
on  hand  to  train  participants,  setup  and  troubleshoot  equipment,  and  review  the 
test  scenario  for  practicality.  It  is  inadequate  and  unrealistic  to  expect  that  the 
researcher  conducting  usability  assessments  on  a  device  will  master  it  in  a  short 
time,  teach  it  to  participants,  and  set  up  and  troubleshoot  it  during  the 
experiment.  The  effort  this  requires  will  inevitably  result  in  negative  compromises 
in  either  the  scope  of  usability  test  or  the  correct  employment  of  the  equipment, 

and  most  likely  in  both  areas.  Convincing  sponsors  and  other  agencies  to  provide 
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proper  support  for  the  equipment  to  be  tested  was  a  major  challenge  during  this 
research  effort,  but  should  be  vigorously  pursued.  During  this  research,  the 
technical  experts  quickly  removed  roadblocks  to  success  that  would  have  been 
otherwise  insurmountable. 

A  domain  expert  may  also  advise  the  usability  researcher  on  the  event 
triggers  and  the  range  of  subsequent  behaviors  likely  to  be  observed.  The 
usability  researcher  must  assiduously  maintain  his  or  her  impartiality  during  this 
relationship.  The  domain  expert  will  often  be  an  advocate  or  developer  of  the 
product  being  tested,  and  their  motivations  and  biases  must  be  recognized  and 
acknowledged  at  every  step  of  the  experiment.  Most  developers  are 
professionals  and  unlikely  to  intentionally  or  maliciously  conceal  negative  aspects 
(or  overly  emphasize  positive  characteristics)  of  their  products.  However,  they 
will  still  hold  assumptions  and  goals  that  may  interfere  with  their  objectivity. 
Corporate  representatives  may  habitually  offer  gifts  or  favors  that  should  politely, 
but  invariably,  be  declined. 

The  demands  of  external  coordination  are  directly  related  to  the  degree  of 
external  validity  sought.  While  frustrating,  time-consuming,  and  generally  outside 
of  researcher’s  area  of  expertise,  bringing  external  realities  into  the  laboratory 
environment  will  enhance  the  credibility  of  the  research  and  the  applicability  of 
the  final  results. 

G.  ANALYTICAL  STRATEGY 

Collected  data  should  be  reviewed  before  analysis.  The  variability 
introduced  by  involving  real  human  participants  and  developmental  stage 
equipment  demands  much  time  be  spent  in  scrubbing  data  for  accuracy  and 
acceptability.  The  validity  of  a  specific  data  point  may  be  questionable  due  to 
improper  data  collection  procedures  (such  as  improper  experimental  design), 
data  entry  errors  (such  as  coding  errors).  Fortunately,  because  most  of  the  data 
collected  will  have  been  recorded  to  video  (potentially  from  multiple  camera 
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angles),  it  is  often  possible  to  review  the  original  data  sources  and  correct  data 
entry  errors. 

It  may  often  be  the  case  that  groups  of  participants  or  specific  equipment 
will  only  be  available  during  limited  time  windows.  Each  piece  of  equipment  to  be 
tested  will  require  participants  spend  training  time  specifically  for  that  equipment, 
so  if  participants  are  to  use  more  than  one  device,  they  will  spend  more  time  in 
training.  Within  these  constraints,  it  may  often  be  difficult  to  arrange  within- 
subjects  experimental  designs.  Worse  yet,  participant  groups  of  convenience 
used  for  a  between-subjects  design,  and  obtained  at  different  times,  may  not  be 
random  samples  of  the  larger  population. 

Within  the  many  practical  constraints,  the  most  likely  experimental  design 
for  military  individual  equipment  usability  testing  is  a  between-subjects  design, 
with  randomized  groups  of  unequal  sample  sizes.  High  between-subject 
variability  should  be  expected.  Potentially,  this  individual  variability  may  be 
accounted  for  using  pre-operation  surveys  of  technical  and  military  experience, 
or  by  testing  the  users  in  some  standard,  related  tasks  to  establish  baseline 
capabilities. 

The  data  gathered  with  a  purely  between-subjects  design  can  be 
compared  with  a  traditional  ANOVA  model  (Figure  6),  or  a  non-parametric  test 
like  the  Kruskal-Wallis  test.  Non-parametric  analysis  provides  protection  against 
non-normality  of  the  data  (a  strong  possibility  in  human  performance  measures) 
and  more  statistical  power  in  data  sets  with  low  numbers  of  participants. 
However,  non-parametric  analysis  may  not  be  possible  with  the  more 
complicated  experimental  designs  (such  as  odd  combinations  of  within-subject 
testing)  that  may  result  from  last-minute  adjustments. 
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Yi,j  -  M  +  ai+  Pj  +  si,j 

where: 

/'  the  device  tested 

j  the  participant  # 

yu  the  task  time  for  participant  j  with  device  /' 
a.t  the  device  effect  for  device  / 
y 3j  the  participant  effect  for  participant  j 

sjJ  random  error  □  n(//,<r) 

Figure  6.  ANOVA  Model  for  Usability  Measures 


The  hypothesis  tested  is  given  in  Table  5.  The  measures  of  performance 
used  will  most  often  be  speed  (task  time)  and  /  or  accuracy  (#  of  errors,  or  error 
rate). 


Table  5.  Research  Hypothesis  for  Equipment  Testing 

Ho:  Users  perform  the  tasks  equally  well  using  the  different 

equipment. 

Hi:  Users  perform  differently  using  the  different  equipment. 


If  the  ANOVA  or  Kruskal-Wallis  tests  detect  a  difference  between  the 
devices,  they  should  be  followed  by  Tukey’s  multiple  comparison  test  or  the 
appropriate  non-parametric  multiple  comparison,  to  then  rank  the  equipment. 
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IV.  CASE  STUDY 


A.  INTRODUCTION  AND  RATIONALE 

The  purpose  of  this  case  study  is  to  demonstrate  the  applicability  and 
validity  of  the  Applied  Warfighter  Ergonomics  Observation  Laboratory  for  usability 
testing  of  military  individual  equipment.  Four  different  Ruggedized  Personal 
Digital  Assistants  (RPDAs)  were  obtained  and  their  hardware  and  software  user 
interfaces  compared.  According  to  the  equipment  developers,  during  their 
research  and  development  phases,  the  RPDAs  have  been  evaluated  primarily 
using  "next-bench  design";  i.e.,  an  engineer  who  most  likely  has  unusually 
sophisticated  technical,  but  no  military  experience,  presents  an  interface  or 
prototype  to  a  similar  engineer  at  the  "next  bench"  over,  who  then  gives  the 
designer  his  or  her  feedback. 

An  additional  goal  of  this  case  study  is  to  serve  as  a  template  for  similar 
studies  in  the  future.  Notes  on  administrative  and  logistical  minutia  are  included 
where  they  might  facilitate  future  studies.  A  tremendous  amount  of  time  was 
spent  during  this  research  correcting  problems  that  might  be  avoided  by  following 
the  recommendations  given  here. 

The  RPDAs  were  provided  by  manufacturers  and  the  associated  military 
program  offices2.  RPDA-D  was  the  first  device  tested,  because  it  was  the  first 
available  in  a  complete  and  testable  configuration.  Initial  support  for  the  device 
was  provided  by  Captain  Francisco  Caceres,  USMC.  Captain  Caceres 
configured  and  provided  three  fully  functional  devices  with  1:250,000  maps  of  the 
Monterey  Peninsula.  Unlike  the  other  single-unit  devices,  RPDA-A  consisted  of  a 
small,  lightweight  handheld  unit,  connected  by  cable  to  a  large  central  unit,  which 
is  further  connected  to  the  communications  unit  with  more  cables.  The 

2  In  this  report,  the  four  devices  tested  are  identified  simply  as  RPDA-A,  RPDA-B,  RPDA- 
C,  and  RPDA-D.  The  actual  device  names  and  manufacturers  are  held  as  For  Official  Use  Only, 
and  can  be  obtained  from  the  Operations  Research  Department  at  the  Naval  Postgraduate 
School. 
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associated  hardware  required  by  RPDA-A  was  carried  in  a  backpack.  Like 
RPDA-A,  RPDA-B  and  RPDA-C  consisted  of  a  single  unit.  Unlike  the  other  units 
and  civilian  PDAs,  RPDA-C  does  not  have  a  pressure-sensitive  screen.  RPDA-C 
uses  a  magnetic  inductive  stylus  and  a  physically  more  durable  screen  than  other 
PDAs.  RPDA-B  used  the  same  hardware  platform  (including  screen  and  stylus) 
as  RPDA-C,  but  with  a  subset  of  the  software  build  used  in  RPDA-A  loaded  onto 
it.  Configured  in  this  manner  solely  for  experimental  testing,  RPDA-B  had  greatly 
reduced  functionality  compared  to  the  other  RPDAs.  Therefore,  RPDA-B  was  not 
tested  using  a  full  tactical  scenario.  Rather,  the  same  group  of  participants 
learned  RPDA-A  and  RPDA-B  together  (because  those  devices  used  similar 
software).  Each  participant  in  this  group  used  both  devices  (A  and  B).  To  counter 
the  order  effects,  a  crossover  design  with  random  assignment  determined 
whether  participants  used  RPDA-A  or  RPDA-B  first. 

B.  METHOD 

1.  Participants 

The  intended  users  of  the  four  different  RPDAs  analyzed  include  U.S. 
Army  Special  Forces,  U.S.  Army  Rangers,  U.S.  Marines,  and  members  of  the 
U.S.  Army  regular  forces.  Participants  for  the  RPDA  studies  were  drawn  from 
Army  and  Marine  Corps  students  in  "casual"  status  (students  not  currently  in 
training  for  various  reasons)  at  the  Defense  Language  Institute  (DLI)  at  the 
Presidio  of  Monterey  (POM).  These  students  are  generally  junior  enlisted 
personnel  in  the  intelligence  fields  awaiting  formal  language  training.  Other 
participants  considered  included  students  at  the  Naval  Postgraduate  School 
(NPS)  and  service  members  from  the  actual  units  receiving  the  RPDAs.  Although 
a  convenient  pool,  NPS  students  were  rejected  since,  as  a  group,  they  are 
entirely  officers  with  high  military  proficiency  and  experience,  and  tend  to  be 
more  sophisticated  computer  users.  Many  attempts  were  made  to  work  with 
actual  members  of  the  units  intended  to  receive  these  devices,  such  as  Army 
Rangers.  Operational  commitments  during  2004-2005,  when  upwards  of  176,000 
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service  members  were  deployed  in  Iraq  for  an  average  of  320  days  each, 
precluded  personnel  being  available  for  academic  research. 

Working  with  actual  participants  from  a  targeted  population  (as  opposed  to 
a  population  of  convenience,  such  as  other  NPS  students,  or  local  civilians), 
introduces  a  number  of  challenges  to  coordination  and  logistics.  For  example, 
one-third  of  the  DLI  students  initially  selected,  trained  on  a  specific  RPDA,  and 
ready  to  be  tested,  were  withdrawn  from  the  study  at  the  last  minute  due  to  other 
demands  on  their  time.  An  entirely  new  group  had  to  be  coordinated  and  trained, 
wasting  weeks  of  effort.  Table  6  breaks  down  the  effort  spent  just  in  collecting  the 
data  in  this  study.  This  estimate  does  not  include  the  initial  time  required  to  learn 
the  lab  equipment  or  any  of  the  later  data  analysis.  The  biggest  time  savings 
could  be  realized  by  arranging  for  somewhat  larger  groups  (8-15  would  be  ideal) 
and  by  having  a  separate  duty  expert  provide  support  for  and  conduct  the 
training  on  the  devices. 


Table  6.  Estimated  Time  Spent  in  Data  Collection 

Activities  per  device  tested 

Est.  Time 

Coordinate  training  and  acquisition  with  owning  agency  4  hrs 

Learn  to  set  up,  operate,  and  troubleshoot  device 

8  hrs 

Configure  and  set  up  lab  for  testing 

1  hr 

Develop,  plan  and  rehearse  training  for  participants 

6  hrs 

Activities  per  testing  group  of  3-10  participants 

Coordination  with  parent  unit  or  agency 

1  hr 

Transportation  and  classroom  logistics 

2  hrs 

Conduct  training  and  practical  application 

3  hrs 

Distribute  and  collect  surveys  and  consent  forms 

0.5  hrs 

Refresher  training  on  testing  day 

0.5  hrs 

Activities  conducted  with  each  participant 

Set  up  lab  equipment  for  this  run 

0.25  hr 

Run  scenario 

0.75  hr 

Coding  each  run 

1.5  hr 

Approximate  total  time  spent 

With  4  devices,  and  4  groups  of  3-6  participants 

120-150  hrs 
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There  were  complications  in  this  study  due  to  high  participant  attrition  and 
reduced  availability  of  those  participants  who  were  able  to  take  part  in  the  entire 
study.  For  example,  the  first  group  of  participants  (using  RPDA-D)  consisted  of 
19  individuals,  including  Soldiers,  Marines,  NCOs,  and  Officers.  After  having 
invested  about  30  hours  in  coordinating  and  conducting  their  training,  the  14 
junior  enlisted  soldiers  were  withdrawn  from  the  study  for  a  work  detail,  leaving 
only  an  officer,  an  NCO,  and  the  Marines. 

2.  Apparatus  and  Instruments 

The  laboratory  equipment  described  in  Part  II  was  used  for  this  case 
study.  Appendix  J  contains  a  detailed  description  of  the  settings  of  each  piece  of 
equipment  and  software  used  during  the  case  study.  More  general  notes  about 
the  setup  follow. 


a.  Recording  Video 

It  is  important  follow  the  checklist  in  Table  4.  Failure  to  do  so 
usually  resulted  in  some  sort  of  problem.  An  “X”  taped  to  the  floor  marked  a 
location  in  the  lab  where  the  participant  would  stand  and  be  in  a  good  view  from 
both  cameras  (front  and  side).  The  ceiling  microphone  was  strung  over  that  spot, 
suspended  approximately  seven  feet  from  the  floor.  Audio  equipment 
(microphone  gain  and  loudspeaker  volume)  were  checked  every  time,  as  these 
settings  are  often  changed  between  participants  or  between  days. 

One  of  the  most  useful  views  of  a  participant  using  a  handheld 
device  is  a  split-screen  shot,  with  the  participant  on  the  left  side  of  the  screen  and 
a  view  of  the  equipment  they  are  using  to  their  right  (see  Figure  7).  To 
accomplish  this,  the  Mobile  Device  Camera  (MDC),  observing  the  RPDA’s 
screen,  was  carefully  adjusted  to  capture  the  RPDA  screen  butted  up  against  the 
extreme  left  side  of  the  camera’s  view.  Then,  the  video  mixer  was  set  (transition 
#7)  to  push  input  3  (from  the  MDC)  across  input  1  (the  front  camera),  just  until 
the  entire  RPDA  screen  was  in  view.  Finally,  the  camera  controller  is  used  to 
adjust  camera  #1  until  the  participant  is  appropriately  positioned  in  the  remaining 
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area.  There  are  two  important  items  when  the  video  mixer  is  set  up  in  this 
fashion.  With  the  T-bar  in  an  intermediate  position:  1)  the  inputs  cannot  be 
changed  (you  cannot  switch  to  camera  2,  for  example),  and  2)  you  must  be  very 
careful  not  to  nudge  the  T-bar,  as  this  will  throw  off  your  split-screen  shot. 


Figure  7.  Participant  and  RPDA  Split-Screen  View 

The  wall-mounted  cameras  were  adjusted  and  presets  for  each 
camera  reset  for  each  individual  participant  to  account  for  differences  in  height. 
This  initial  setup  took  about  five  minutes  per  participant.  Each  recording  session 
began  with  the  participants  stating  their  participant  ID  number,  to  facilitate 
organizing  the  videos  later. 

3.  Procedure 

During  Tactical  Network  Topology  field  experiments  at  Camp  Roberts, 
interviews  and  focus  groups  were  conducted  to  determine  how  users  perceive 
the  usability  of  military  PDAs,  their  current  and  potential  functionality,  and  what 
barriers  exist  to  adoption.  A  functional  task  analysis  of  users  operating  RPDAs 
was  conducted.  The  results  of  the  functional  task  analysis  lead  to  the 
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development  of  the  scenario  and  the  specific  tasks,  conditions,  and  standards 
with  which  to  perform  the  usability  testing. 

The  RPDAs  were  compared  using  a  between-subjects  experimental 
design  under  laboratory  conditions.  RPDA-D  was  the  first  device  tested.  A  group 
of  19  participants,  including  staff  and  students  in  casual  status  at  DLI,  were 
trained  by  Captain  Koichi  Takagi  USMC,  1LT  Derek  Read,  USAF,  and  1LT  Matt 
Simpson,  USAF,  in  a  classroom  on  the  Presidio  of  Monterey.  All  of  the  classroom 
instruction  was  recorded  and  is  available  on  DVD  from  the  Applied  Warfighter 
Ergonomics  Center  at  the  Naval  Postgraduate  School.  First,  a  description  of  the 
research  was  presented  (see  Appendix  H),  and  administrative  details  were 
addressed.  Using  the  Noldus  Mobile  Device  camera  attached  to  an  RPDA-D  unit, 
Captain  Takagi  conducted  a  fifteen-minute  class  on  the  functionality  and 
procedures  for  operating  RPDA-D.  After  that,  the  students  paired  off  and 
practiced  the  user  tasks  using  the  Monterey  peninsula  maps  loaded  onto  RPDA- 
D,  and  the  practical  application  test  sheet  given  in  Appendix  G,  section  5.  While 
waiting  to  use  the  few  RPDA  units  available,  the  rest  of  the  students  practiced 
their  conventional  skills  using  paper  maps,  protractors,  and  the  same  points  on 
the  map  to  be  used  in  the  test  scenarios. 

The  primary  hypothesis  tested  using  this  procedure  is  shown  in  Table  7. 


Table  7.  Hypothesis  Hi:  Differences  between  RPDAs 

H0:  Users  perform  the  tasks  equally  well  using  the  different 

RPDAs. 

Hi:  Users  perform  differently  using  the  different  RPDAs. 


a.  Interviews  and  Focus  Groups 

An  ad-hoc,  non-attributional  military  PDA  focus  group  was 
conducted  1600  -  1730  on  22  Feb  2005,  at  Camp  Roberts,  CA  during  a  Tactical 
Network  Topology  field  exercise.  The  participants  were:  three  senior  Special 
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Forces  staff  NCOs  participating  in  a  military  RPDA  evaluation  exercise,  one 
representative  from  USASOC  program  office  (but  not  the  Mobile  Access 
Infrastructure  program  office  responsible  for  military  PDAs),  and  one  employee  of 
an  RPDA  development  company. 

The  goal  was  to  determine  possible  requirements  and  capabilities 
for  handheld  military  computing  devices  (military  PDAs)  in  a  “brainstorming” 
environment.  Ideas  were  encouraged  regardless  of  current  feasibility  or  real 
applicability.  Appendix  J  contains  a  free-form  description  (but  not  a  verbatim 
transcript)  of  the  topics  discussed,  including  possible  operational  requirements 
for  military  PDAs,  general  and  specific  functionality  that  military  PDAs  might 
have,  possible  uses  for  and  tactical  employment  of  military  PDAs,  and  random 
comments.  Some  participants  had  not  used  military  or  civilian  PDAs  to  any  great 
extent,  some  were  unfamiliar  with  the  military  tasks  discussed.  Non-attribution 
was  emphasized,  and  it  was  understood  that  the  discussion  would  not  generate 
obligations  by  the  contractor  or  the  government. 

b.  Surveys 

The  pilot  surveys  were  given  to  other  NPS  students  and  evaluated 
by  Dr.  Nita  Lewis  Miller  and  Dr.  Jeff  Crowson  during  the  OA3402  Research 
Methods  course.  Over  a  several  month  period,  several  revisions  of  the  pilot 
surveys  were  evaluated  for  brevity,  clarity,  and  relevance.  As  a  result  of  the  pilot 
surveys,  questions  were  rewritten,  response  anchors  revised,  the  format  was 
cleaned  up,  and  several  questions  were  discarded.  The  final  form  of  surveys 
captures  the  participant  responses  with  minimal  demands  on  the  participant  (for 
time  and  recall)  and  the  researchers  (for  coding  and  interpretation).  Due  to 
discrepancies  in  how  questions  were  interpreted  between  participants,  the 
wording  of  several  questions  was  slightly  modified  between  groups.  In  some 
cases,  this  precludes  direct  comparison  of  the  survey  results  between  the 
groups.  For  example,  question  number  22  on  the  Technology  Experience  Survey 
originally  read,  “I  send  and  receive  email  messages _ times  per  week.”  This 

question  was  answered  consistently  during  the  pilot  surveys,  where  the 
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administrator  was  available  for  questions.  The  participants  during  the  case  study 
took  the  surveys  home.  Some  answered  in  terms  of  number  of  email  sessions, 
and  some  in  terms  of  individual  messages.  The  question,  and  all  similar 

questions,  were  changed  to  read  “I  send  and  receive  email  messages _ 

hours  per  week.”  Although  this  renders  the  actual  data  from  those  survey 
questions  invalid,  the  survey  is  now  more  robust  for  use  in  future  studies. 

Organization  of  paperwork  required  for  this  research  was  facilitated 
with  a  portable  filing  system  using  separate,  color-coded  manila  folders  labeled 
as  shown  in  Table  8. 


Table  8.  Surveys  and  Related  Paperwork 

1.  Prepared  packets  (paper-clipped  packets  to  hand  out  to 
participants,  containing  all  the  necessary  blank  forms  2-6 
below) 

2.  Participant  Information  Forms 

3.  Maps  to  NPS 

4.  Blank  Participant  ID  Forms  and  Participant  ID  Cards 

5.  Blank  Consent  Forms  (3  pages) 

6.  Blank  Technology  Experience  Surveys 

7.  Blank  Handheld  Device  Evaluations 

8.  Briefs  and  Lesson  Plans 

9.  Teaching  Materials  and  Practical  Application  Handouts 

10.  Scenario  Maps  and  Instructions 

1 1 .  Completed  Consent  Forms 

12.  Completed  Participant  ID  Forms 

13.  Completed  Technology  Experience  Surveys 

14.  Completed  Handheld  Device  Evaluations 


Pre-operation  Surveys.  Users  completed  the  Technology 
Experience  Survey  (Appendix  E)  before  using  the  equipment  to  assess  their 
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demographic  background,  relevant  technical  experience,  and  relevant  military 
skills.  Although  it  had  been  administered  in  several  previous  pilot  studies,  during 
this  study,  the  survey  was  adjusted  in  several  ways  as  described  above.  Some 
questions  therefore  cannot  be  directly  compared  between  users. 

Post-operation  Surveys.  Users  completed  the  Handheld  Device 
Evaluation  (Appendix  E)  after  using  the  equipment  to  obtain  quantitative  and 
qualitative  feedback  from  the  user  on  the  devices.  The  survey  results  were  used 
in  conjunction  with  the  usability  testing  to  determine  how  well  users'  impressions 
of  their  performance  and  satisfaction  with  the  equipment  corresponds  to  their 
measured  performance.  The  primary  measures  of  interest  on  this  survey  are 
overall  satisfaction  with  the  device,  and  a  rating  of  ease-of-use.  Many  questions 
on  this  survey  are  intended  to  be  broadly  applicable  to  field  use  of  RPDAs  and 
other  devices,  and  so  elicited  “Not  Applicable”  responses  in  the  laboratory 
testing. 


c.  Usability  Observations 

Tasks.  The  basic  tasks  listed  in  Table  9  were  chosen  as  being 
tactically  relevant,  familiar  to  typical  users,  easily  taught  in  a  brief  session, 
performed  by  both  current  technology  and  RPDAs,  and  common  to  all  of  the 
devices  studied.  Table  10  and  Table  11  list  tactical  tasks  that  were  identified  as 
being  of  possible  interest  to  similar  studies. 


Table  9.  Basic  Tactical  Tasks  Used  in  this  Study 

1 .  Perform  a  communications  check. 

2.  Plot  a  route  with  two  legs. 

3.  Read  three  grid  coordinates  from  a  map. 

4.  Determine  the  bearing  and  range  from  one  point  to  another, 
and  the  total  distance  of  a  route. 

5.  Send  a  message. 

6.  Receive  a  message. 

7.  Retrieve  and  read  back  a  previous  message. 
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Table  10.  Tasks  considered,  but  rejected  from  this  study 

1 .  Interpret  streaming  video  data.  This  task  was  rejected  as  not 
being  performed  by  existing  technology,  and  not  immediately 
familiar  to  typical  users 

2.  Send  /  receive  digital  voice  messages  (Voice  over  IP  or 
other  technology).  This  task  was  rejected  as  not  being 
implemented  by  all  RPDAs,  and  as  unlikely  to  be  a  major 
decision  criteria  for  any  RPDA.  Users  were  emphatic  that 
they  would  prefer  to  use  existing  voice  communication 
radios.  RPDA  digital  communication  links  may  not  be  well- 
suited  for  critical  voice  communications  for  because  of  the 
propagation  characteristics  of  the  radio  frequencies  used. 

3.  Send  /  receive  sophisticated  map  overlays  or  multimedia 
files.  Although  of  considerable  tactical  relevance  and 
interest,  this  task  was  rejected  as  being  overly  complicated 
to  teach  to  users. 


Table  1 1 .  Tasks  for  consideration  in  future  studies 

1 .  Receive  a  situational  update,  and  brief  it  to  subordinates. 

2.  Create  a  tactical  map  overlay. 

3.  Send  a  map  overlay. 

4.  Observe  a  video  or  situation  and  send  an  intelligence  report. 

5.  Make  a  tactical  assessment  based  on  tactical  images  or 
video  received. 

6.  Receive  and  process  intelligence  reports. 

7.  Establish  a  (currently  non-existing)  communications  link. 

8.  Send  standard  tactical  messages  (medevac,  nine-line  brief, 
call-for-fire,  etc.) 

9.  Send  administrative  or  logistic  messages. 

The  users  performed  these  tasks  both  conventionally,  (for  example, 
reading  a  standard  paper  military  1:50,000  scale  map),  and  using  alternate 
means;  in  this  case,  using  an  RPDA.  The  research  hypothesis  of  this  within- 
subjects  experimental  design  is  shown  in  Table  12.  During  the  course  of  the 
study,  it  was  found  that  users  performed  the  tasks  in  very  different  ways, 
requiring  several  individual  navigation  tasks  to  be  combined  into  a  single  larger 


50 


measurement  of  mapping  skill.  Therefore,  the  conventional  and  alternate  tasks 
no  longer  corresponded  well  with  each  other,  and  the  hypothesis  presented  in 
Table  12  could  not  be  directly  tested.  To  address  hypothesis  H2,  the  test  scenario 
must  be  designed  in  tight  conjunction  with  the  device  capabilities  and  tuned  in 
pilot  studies  to  ensure  that  the  tasks  correspond  well  between  the  conventional 
and  alternate  means.  The  seven  chosen  tactical  tasks  were  coded  for  data  entry 
as  shown  in  Table  13. 


Table  12.  Hypothesis  H2:  Conventional  vs.  Alternate  Means 

Ho:  Users  perform  the  tasks  equally  well  using  conventional  and 

alternate  means. 

H2:  Users  perform  differently  using  conventional  and  alternate 

means. 


Table  13.  Coded  tactical  tasks 


Task# 

Description 

Means 

Abbreviated 

Description 

1 

Perform  a  communications  check 

Conv. 

Alt. 

1c  comm  chk 
la  comm  chk 

2 

Plot  a  route  with  two  legs 

Conv. 

Alt. 

2c  plot  rte 

2a  plot  rte 

3 

Determine  the  grid  coordinates  of  three 
locations  from  a  map 

Conv. 

Alt. 

3c  read  grid 

3a  read  grid 

4 

Determine  the  bearing  and  range  from 
one  point  to  another,  and  the  total 
distance  of  a  route  (when  supported) 

Conv. 

Alt. 

4c  brg  mg 

4a  brg  rng 

5 

Send  a  message 

Conv. 

Alt. 

5c  send  msg 

5a  send  msg 

6 

Receive  a  message 

Conv. 

Alt. 

6c  rev  msg 

6a  rev  msg 

7 

Retrieve  and  read  a  previous  message 

Conv. 

Alt. 

7c  read  back 

7a  read  back 

As  the  testing  progressed,  inconsistencies  were  noted  that  required  the 


task  definitions  and  scoring  standards  be  re-defined.  For  example,  in  the  first  part 

of  the  scenario,  participants  were  instructed,  in  three  separate  steps,  to  plot  a 
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route,  to  read  three  grids,  and  to  compute  a  bearing  and  range.  Although  the  first 
group  of  participants  performed  the  task  consistently,  subsequent  participants 
would  sometimes  calculate  all  the  required  information  in  the  first  step,  so  that 
the  task  time  for  the  first  step  was  much  longer,  and  the  subsequent  task  times 
were  virtually  zero.  The  task  was  then  re-defined  and  the  observations  were  re¬ 
scored  to  collect  all  the  task  time  of  the  three  steps  into  a  single  task,  whether 
they  had  broken  it  up  or  not.  Although  some  granularity  is  lost  in  doing  this,  the 
task  can  now  be  reliably  compared  between  participants. 

Another  problem  arose  only  after  observing  many  participants.  During  the 
initial  and  refresher  training  and  practical  application,  about  ten  points  were 
identified  on  the  paper  and  PDA  maps  including  all  six  of  the  points  used  in  the 
scenario.  During  the  testing  phase,  some  participants  were  still  confused  about 
some  of  the  locations,  especially  the  Monterey  Airport.  This  confusion  resulted  in 
extremely  long  task  times  as  the  participants  scrolled  around  the  PDA  screens  in 
a  fruitless  search.  When  directed  to  the  location,  they  were  able  to  complete  the 
task.  These  data  points  are  outliers  in  the  sense  that  the  are  the  sum  of  two 
behaviors:  tactical  skill  task  time  (the  behavior  of  interest),  and  search  time  (an 
error  term).  Rather  than  discard  them  completely,  the  navigational  tasks  were 
redefined  as  "off-task"  after  the  participant  had  been  subjectively  judged  as  "lost" 
for  more  than  five  seconds.  Indications  of  lost  behavior  include  looking  directly  at 
the  target  and  then  away  from  it,  and  searching  in  an  entirely  wrong  direction  for 
more  than  five  seconds.  Subjective  measures  of  off-task  behavior  such  as  these 
should  be  avoided  in  future  studies  by  fully  inoculating  participants  against 
confusion  during  tasks.  In  this  case,  more  explicitly  training  participants  to  the 
locations  of  interest  may  have  helped.  Inter-coder  and  intra-coder  reliability 
analysis  was  used  to  ensure  that  the  subjective  correction  applied  (defining  lost 
behavior  as  off-task)  was  applied  consistently. 

Other  observed  behaviors:  errors  and  posture.  In  addition  to  the 
tasks  observed,  the  five  categories  of  error  listed  in  Table  14  were  coded  as  they 
occurred. 
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Table  14.  Errors  by  Type 


Error 

Description 

Abbreviated 

Description 

1 

Slip:  unintended  action,  user  realizes  error 

slip 

2 

Mistake:  intended  action,  user  is  unaware  of  error 

mistake 

3 

Device  error:  the  device  fails  or  crashes,  not  due 
to  user  action 

device  err 

4 

Interface  error:  user  has  a  problem  with  device 
input  or  output,  such  as  getting  device  to  register 
input,  or  reading  labels  or  screens 

Interface  err 

5 

Equipment  handling  error:  due  to  using  this 
device,  the  user  spends  time  correcting 
equipment  or  has  a  physical  difficulty  handling  or 
operating  any  piece  of  gear,  such  as  dropping 
rifle,  untangling  cords,  etc. 

eq  handle  err 

The  tactical  postures  described  in  Table  15  were  coded  as  they  occurred. 


The  specific  criteria  for  a  tactical  (vice  administrative)  posture  was:  shooting  hand 
on  the  pistol  grip  of  the  weapon  and  able  to  engage  in  less  than  one  second, 
otherwise  define  state  as  admin  posture. 


Table  15.  Tactical  Postures 


Tactical 

Posture 

Description 

Abbreviated 

Description 

1 

Tactical:  The  participant  is  immediately  ready  for 
a  combat  situation,  shooting  hand  on  the  pistol 
grip  of  the  weapon  and  able  to  engage  in  less 
than  one  second 

Tactical 

2 

Administrative:  The  participant  is  not  immediately 
ready  for  a  combat  situation,  for  example,  with  his 
weapon  slung  over  his  shoulder. 

Admin 

The  conventional  segment  of  the  scenario  was  identical  for  each  RPDA. 


The  RPDA  segment  was  intended  to  be  identical  for  each  RPDA.  Some  RPDAs 
were  not  available  at  the  time  the  scenario  was  written,  and  the  scenario  was 
adjusted  slightly  to  accommodate  different  capabilities.  This  prevented  between- 
device  comparisons  in  those  specific  tasks,  but  the  majority  of  tasks  can  still  be 
compared  between  devices.  The  tasks  that  had  to  be  modified  can  still  be 
compared  to  the  baseline  performance  using  conventional  equipment. 
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Conditions.  Laboratory  testing  took  place  at  the  Naval 
Postgraduate  School,  in  the  Applied  Warfighter  Ergonomics  laboratory  in 
Glasgow  Hall  room  221.  Environmental  conditions  and  personal  characteristics  of 
the  participants  not  explicitly  listed,  were  not  measured.  Environmental 
conditions,  such  as  lighting  and  temperature,  were  constant,  comfortable,  and 
typical  of  an  office  environment.  All  testing  occurred  between  8:00  am  and  6:00 
pm  on  weekdays. 

Test  participants  wore  their  own  camouflage  utility  uniforms  and 
the  older  model  combat  equipments  listed  in  Table  16. 


Table  16.  Battle  Gear  Worn  in  the  Tactical  Scenario 


Item  # 

Description 

Abbreviated 

Description 

1 

Kevlar  helmet 

helmet 

2 

Kevlar  flak  vest 

flak 

3 

Load-bearing  vest  with  magazine  pouches  and 
canteens 

LBV 

4 

Rubber  M-16A2  rifle 

weapon 

Standards:  Measures  of  Effectiveness  and  Measures  of 
Performance.  Each  task  has  a  precisely  defined  measure  of  success  defined  in 
Table  17.  All  tasks  start  as  soon  as  the  verbally  issued  instructions  are 
completed,  for  the  first  time,  if  the  instructions  are  repeated.  If  the  instructions 
were  completely  unintelligible  to  the  participants,  then  task  start  is  when  the 
instructions  are  issued  comprehensibly. 


3  Individual  equipment  was  temporarily  loaned  from  the  Marine  Corps  Detachment  at  the 
Defense  Language  Institute.  Considerable  time  was  spent  unsuccessfully  attempting  to  obtain 
current-issue  equipment. 
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Table  17.  Measures  of  Task  Success 


Task 

Measure  of  success 

comm  chk 

Participant  verifies  that  2-way  communication  is  established  by  sending 
and  receiving  a  simple  message. 

Task  ends:  participant  completes  saying  or  typing  “comm  check” 

plot  rte 

Participant  performs  a  basic  land  navigation  task,  including  plotting  a 
route  with  two  legs,  recording  the  three  grid  coordinates  (endpoints  and 
middle  waypoint),  computing  the  bearing  and  range  of  the  first  leg,  and 
the  total  distance  of  the  route. 

Task  ends:  participant  begins  reading  the  completed  information. 
Recorded  as  “off-task”  while  actually  reading  information  out  loud. 

Off-task  when  participant  has  been  "lost"  on  map  for  five  seconds,  and 
scrolling  around  only  because  they  cannot  find  a  location. 

read  grid 

Participant  computes  a  six-digit  grid  coordinate  within  plus  or  minus  500 
meters.  This  large  leeway  is  due  to  imprecision  in  identifying  certain 
landmarks  on  the  military  topographical  maps. 

Task  ends:  participant  begins  reading  the  completed  information.  Off-task 
when  participant  has  been  "lost"  on  map  for  five  seconds,  and  scrolling 
around  only  because  they  cannot  find  a  location. 

brg  rng 

Participant  computes  a  bearing  within  plus  or  minus  2  degrees  and  a 
range  within  plus  or  minus  500  meters.  Total  distance  also  will  have  an 
accuracy  of  plus  or  minus  500  meters  or  5%,  whichever  is  higher. 

Task  ends:  participant  begins  reading  the  completed  information.  Off-task 
when  participant  has  been  "lost"  on  map  for  five  seconds,  and  scrolling 
around  only  because  they  cannot  find  a  location. 

send  msg 

Participant  sends  a  message  that  would  be  unambiguously  understood 
by  another  military  member  in  the  current  context.  Abbreviations  and 
tactical  terminology  are  acceptable. 

Task  ends:  participant  completes  reading  the  message  out  loud. 

rev  msg 

Participant  demonstrates  that  he  or  she  understands  a  message  by  either 
performing  an  instruction  or  reading  it  aloud  to  the  notional  patrol 
members.  Paraphrasing  is  acceptable  if  the  precise  meaning  is  retained. 

Task  ends:  participant  indicates  that  they  have  understood  the  message 
by  performing  an  appropriate  action,  or  when  they  have  completed 
copying  down  information. 

read  back 

Participant  locates  and  reads  back  a  message  previously  sent  or  received 
that  he  or  she  would  not  likely  be  able  to  recall  from  memory. 

Task  ends:  participant  begins  reading  the  message  out  loud. 

The  metrics  used  to  evaluate  behaviors  were  task  time, 


number  of  errors  by  type,  and  tactical  posture.  Task  time  began  when  the 


55 


scenario  narrator  finished  reading  the  task  (the  first  time,  if  a  long  instruction  was 
repeated).  Task  time  ended  (that  is,  the  participant  was  coded  as  "off-task") 
when  the  participant  began  the  next  behavior,  which  was  usually  reading 
something  (like  a  grid  coordinate)  aloud  to  his  notional  patrol  members.  There 
were  occasions  when  task  completion  times  is  in  doubt,  usually  because  the 
participant  appears  to  have  cognitively  moved  off  the  task,  but  forgot  to  perform 
the  ending  behavior.  For  example,  when  reading  a  message  on  the  RPDA,  the 
task  ending  behavior  is  to  read  the  message  aloud  to  the  notional  patrol 
members.  Occasionally,  the  participant  would  read  the  message  but  not  read  it 
aloud,  although  he  or  she  clearly  had  understood  the  message  and  would  then 
read  it  aloud  when  prompted.  A  “best-guess”  in  these  situations  would  mark  the 
end  of  task  time. 

Another  ambiguous  behavior  to  code  was  in  exercising  paper 
mapping  skills.  When  given  points  on  a  map,  some  participants  simply  wrote 
down  the  points  and  awaited  further  instructions  (the  desired  behavior).  The  next 
tasks  were  to  plot  the  points  and  read  the  grid,  and  then  to  plot  the  route  and 
read  the  bearing  and  range  of  just  the  first  leg.  Some  participants  performed  all 
the  tasks,  as  well  as  computing  the  bearing  and  range  of  all  legs  (rather  than  just 
the  first),  in  a  single  step.  This  made  it  impossible  to  separate  out  task  times  into 
times  for  performing  each  of  the  individual  tasks.  Therefore,  the  routine  originally 
envisioned  as  three  separate  tasks  (plot  route,  read  grid,  compute  bearing  and 
range),  was  collapsed  into  a  single  longer  task  (plot  route),  with  the  task  time 
starting  and  stopping  when  the  participant  was  reading  the  data  or  receiving 
instructions. 

Scenario.  The  scripts  for  the  tactical  scenarios  presented  are 
detailed  in  Appendix  F.  The  scenario  used  a  second  computer  running  a  tactical 
decision-making  game  presented  to  the  test  participant  on  a  52”  plasma  screen 
with  stereo  speakers.  The  software  used  is  called  Combat  Decision  Range 
(CDR),  and  developed  by  the  Marine  Corps  Warfighting  Lab.  The  CDR  was  run 
on  a  laptop  computer  with  S-video  output  driving  the  plasma  screen  in  the  other 
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room.  The  S-video  output  was  also  split  off  to  input  four  of  the  video  mixer  for 
monitoring  from  the  Observer  room. 

Combat  Decision  Ranges  (CDRs)  were  designed  to  present  tactical 
scenarios  to  squad  and  platoon  leaders  in  the  Marine  Corps.  The  CDRs  present 
stressful  situations  demanding  quick  thinking  by  the  participant.  Within  this  study, 
the  CDR  was  used  to  build  a  combat  mindset  in  the  participants.  The  tasks 
triggered  for  the  usability  study  were  very  specific,  and  the  participant  was  not 
allowed  freedom  of  decisions  such  as  the  CDR  was  designed  for.  Therefore, 
there  was  an  artificiality  to  the  test  environment,  as  the  scenario  was  interrupted 
to  issue  specific  instructions  to  the  participant.  The  participants  were  briefed  in 
advance  that  the  tasks  were  at  times  tactically  illogical  or  inconsistent  with  other 
parts  of  the  scenario. 

In  the  scenario  designed  for  this  study,  seven  tasks  were  triggered 
twice,  once  using  conventional  means  and  once  using  the  RPDA  (the  alternate 
means).  The  key  tasks  were  triggered  more  than  once  in  the  conventional  and 
alternate  means.  Repeating  exactly  the  same  scenario  twice  would  have 
introduced  learning  effects  into  the  second  performance  of  the  task.  Therefore, 
the  situations  for  triggering  a  task,  such  as,  “send  a  short  message”,  were  similar, 
but  not  identical,  during  the  conventional  and  alternate  phases  of  the  scenario. 
Only  one  complete  scenario,  with  conventional  and  alternate  segments,  was 
administered  to  the  participant,  and  all  participants  experienced  the  exact  same 
sequence  of  events. 

Conducting  the  scenario  and  observation  required  the  operation  of 
the  computer  playing  the  scenario  videos,  operating  the  camera  controller,  video 
mixer,  and  computer  recording  the  video,  reading  aloud  the  participant 
instructions  over  the  microphone,  and  talking  on  a  handheld  radio  to  the 
participant  when  appropriate  in  the  scenario.  Additionally,  the  participants  were 
briefed,  and  assisted  with  donning  their  equipment  and  operating  the  RPDA.  Due 
to  all  of  these  issues,  a  research  assistant  was  needed  for  all  recording  sessions. 
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C.  CASE  STUDY  RESULTS 

1.  Interviews  and  Focus  Groups 

Interviews  and  focus  group  discussions  (details  in  Appendix  K)  with  the 
developers  and  program  managers  of  the  RPDAs  revealed  several  assumptions 
frequently  are  made  about  the  population  of  users: 

1.  Most  users  are  comfortable  with  the  Microsoft  Windows  user  interface 
model. 

2.  Most  users  are  frequent  video  game  players  (sometimes  referred  to  as 
the  “Nintendo  generation”). 

3.  Most  users  have  some  degree  of  technical  knowledge  about 
computers  and  networking. 

4.  Most  users  have  and  operate  cell  phones  or  civilian  PDAs  on  a  regular 
basis. 

As  shown  in  the  next  section,  the  survey  results  in  this  study  have  not 
borne  out  these  assumptions  as  being  entirely  consistent  with  our  test  user  base. 
Direct  surveys  of  the  actual  user  population  of  these  RPDAs  have  not  been 
conducted. 

2.  Survey  Data 

The  individual  and  aggregate  raw  survey  results  can  be  found  in  Appendix 
J.  Several  of  the  surveys  were  not  turned  in  due  to  participants  forgetting  them  or 
not  having  time  to  fill  out  the  post-operation  survey.  Some  results  from  the  pre¬ 
operation  survey  about  the  participants  in  this  study  include: 

36%  do  not  have  a  cell  phone  (5  hr/wk  median  talk  time  for 
participants  who  have  a  cell  phone), 

50%  have  never  operated  PDA  before, 

54%  have  a  computer  at  home, 

Median  age  was  20  yrs  old, 

84%  directly  from  boot  camp, 
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Participants  did  not  comment  as  extensively  on  the  surveys  as  was 
hoped. 


3.  Usability  Observations 

Observations  were  coded  in  the  Observer  software.  The  Observer’s 
elementary  statistics  functionality  was  used  to  create  a  file  containing  the  total 
durations  of  all  tasks,  by  individual  participant  and  coder.  This  data  file  was 
exported  to  Microsoft  Excel,  where  it  was  formatted  into  comma-separated  value 
(.CSV)  data  files  for  analysis  in  standard  statistics  software.  S-Plus  version  7.0.3 
(Insightful  Corporation,  2005)  was  used  for  all  statistical  analysis,  as  described 
below. 

rpda.orig  is  the  unedited  S-Plus  data  file.  It  contained  some 
questionable  data  points:  Due  to  time  constraints,  participant  504  experienced  an 
abbreviated  scenario  that  was  not  similar  to  the  conditions  used  in  the  rest  of  the 
testing.  Some  participants  tested  two  devices,  in  two  complete,  separate 
scenarios.  A  large,  advantageous  learning  effect  was  presumed  for  the  second 
device  tested. 

As  shown  in  Table  18,  the  S-Plus  data  frame  rpda  removed  the 
questionable  data  points  from  rpda.orig.  The  measurements  for  devices 
tested  second  were  deleted.  Participant  504’s  data  were  deleted. 
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Table  18.  The  S-Plus  data  frame  rpda 

>  rpda=rpda . orig [ (rpda . orig$order==l)  & 

(rpda.orig$id  !=  504),] 

>  attach (rpda) 

>  dim (rpda) 

[1]  10  114 


> 

rpda  [ 

,0: 

12] 

# 

a 

sample  < 

of 

what 

the 

data 

looks  like 

id 

dev 

order 

zl 

z2 

z3 

z4 

z5 

z  6 

z7 

tl 

t2 

2 

502 

1 

1 

8 

.0 

264 

40 

.0 

78 

.0 

15 

.0 

30 

14 

.0 

74 

NA 

5 

505 

1 

1 

14 

.5 

290 

31 

.0 

112 

.5 

15 

.0 

44 

6 

.0 

40 

NA 

6 

506 

1 

1 

14 

.0 

321 

29 

.0 

111 

.0 

23 

.0 

51 

18 

.0 

26 

NA 

7 

501 

2 

1 

5 

.0 

502 

100 

.0 

89 

.0 

5 

.0 

52 

3 

.0 

NA 

NA 

9 

503 

2 

1 

16 

.0 

318 

23 

.0 

49 

.0 

25 

.5 

32 

32 

.5 

NA 

NA 

13 

513 

4 

1 

30 

.5 

307 

32 

.0 

80 

.0 

15 

.5 

21 

12 

.0 

22 

55 

14 

514 

4 

1 

11 

.5 

258 

12 

.0 

85 

.5 

36 

.5 

43 

11 

.0 

22 

46 

15 

515 

4 

1 

15 

.0 

286 

31 

.0 

84 

.0 

15 

.0 

18 

5 

.0 

2 

92 

16 

527 

4 

1 

10 

.0 

189 

2 

.5 

37 

.5 

63 

.5 

29 

3 

.0 

1 

46 

17 

528 

4 

1 

7 

.0 

231 

5 

.0 

38 

.0 

18 

.0 

27 

1 

.0 

1 
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Measurements  were  times  (in  seconds)  to  complete  a  specific  military 
task.  There  were  seven  tasks,  as  described  in  Table  13.  Conventional  tasks  were 
denoted  tasks  zi  through  z7.  Tasks  performed  on  the  RPDAs  were  denoted  ti 
through  t7.  The  tasks  were  generalized  into  mapping  skills  tasks  (t2  -  t4)  and 
communications  tasks  (t-i,  and  t5  - 17). 

A  device  may  perform  well  on  some  tasks,  but  poorly  on  others.  In  some 
cases,  it  was  possible  to  move  functionality  from  one  device  to  another,  to  create 
a  new  device  that  performs  some  tasks  like  RPDA-A,  and  other  tasks  like  RPDA- 
D.  Therefore,  the  best  performance  in  each  task  individually  was  of  interest, 
rather  than  an  overall  rating  of  the  devices.  Using  separate  evaluations  for  each 
task  also  allowed  decision  makers  to  decide  for  themselves  which  were  the  most 
important  tasks. 

Tasks  2,  3,  and  4  were  very  closely  related  mapping  skills  tasks.  The 
conventional  measurements  were  treated  as  three  separate  measurements  of 
mapping  skills,  and  combined  into  a  single  mapping  skills  score.  They  were  all 
adjusted  to  the  scale  of  conventional  task  2  (the  z2  measurements),  and 
averaged  as  shown  in  Figure  8: 
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mapskill  =  — 


\ 


J 


>  rpda$mapskill  =  (z2  +  median (z2,T)  /  median (z3,T)  *  z3 

+  median (z2,T)  /  median (z4,T)  *  z4  )  /3 


Figure  8.  Mapskill  Function  (algebraically  and  S-Plus  code) 


The  medians,  rather  than  the  means,  were  used  for  scaling,  as  the 
measured  task  times  have  heavy  right  tails.  The  participants  with  the  longest  task 
times  also  had  the  greatest  variability  in  time,  and  the  median  was  chosen  as  a 
more  stable  estimate  of  central  value. 

The  mapskill  score  was  used  as  a  composite  measure  of  an  individual’s 
conventional  mapping  ability.  It  had  the  advantage  of  incorporating  three 
separate  measurements  from  an  individual.  The  graph  below,  and  the 
regressions  following  it,  show  how  the  mapskill  score  was  related  to  the 
individual  Z2,  Z3,  and  z4  measurements,  and  how  those  skills  were  related  to  each 
other. 


61 


>  pairs (rpda[ ,c ("mapskill" , "z2" , "z3" ,"z4") ] ,pch=16 , cex=l) 
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>  summary ( lm ( z2~mapskill ,  data=rpda) ) 

Multiple  R-Squared:  0.888 

F-statistic:  p-value  is  0.000 

>  summary ( lm ( z3~mapskill ,  data=rpda) ) 

Multiple  R-Squared:  0.946 

F-statistic:  p-value  is  0.000 

>  summary ( lm ( z4~mapskill ,  data=rpda) ) 

Multiple  R-Squared:  0.398 

F-statistic:  p-value  is  0.050 

Figure  9.  Pairs  plots  of  mapskill,  z2,  z3,  and  Z4 


Participant  skill  in  mapping  tasks  on  an  RPDA  was  expected  to  be  partially 
related  to  their  skill  in  performing  similar  tasks  using  conventional  means.  The 
plots  below  show  that  there  may  be  some  component  of  mapskill  in  the 
measured  task  times  z2,  z3,  and  z4.  These  plots  also  show  that  the  tasks 
themselves  may  be  independent  of  each  other,  and  also  that  there  were  some 
measurements  with  extremely  long  task  times,  relative  to  the  other  tasks. 
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If  a  statistically  significant  difference  is  found  between  the  devices,  the 
cause  of  the  difference  may  be  that  one  device  is  actual  better  than  another.  The 
most  important  reason  a  difference  might  be  seen,  other  than  due  to  the  different 
devices,  is  that  participants  are  different  from  each  other.  If  the  groups  testing 
each  device  appear  to  be  dissimilar  due  to  small  sample  size  or  inspection  of  the 
participants,  an  adjustment  must  be  made  for  participant  effects. 

A  naive  heuristic  for  adjusting  for  this  participant  effect  is  to  form  a  ratio  of 
the  RPDA  task  time  to  the  mapping  skill  score  (which  are  both  times  in  seconds). 
The  results  of  this  study  do  not  directly  support  this  transformation,  however,  it  is 
proposed  as  a  way  to  move  forward  with  the  evaluation  while  adjusting  in  some 
way  for  participant  effect. 

>  rpda$t2adj  =  t2  /  mapskill 

>  rpda$t3adj  =  t3  /  mapskill 

>  rpda$t4adj  =  t4  /  mapskill 
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Unlike  the  closely  related  conventional  navigation  tasks  (z2,  z3,  and  z4), 
which  could  be  viewed  as  separate  measurements  of  more  general  mapping 
skill,  the  communications  tasks  were  less  like  each  other.  It  is  not  clear  that  a 
single  skill  score  is  appropriate  with  them.  Task  times  for  the  communications 
tasks  are  therefore  adjusted  directly  from  a  ratio  with  the  original  task. 

>  rpda$tladj  =  tl  /  zl 

>  rpda$t5adj  =  t5  /  z5 

>  rpda$t6adj  =  t6  /  z6 

>  rpda$t7adj  =  t7  /  z7 


Several  statistical  tests  for  a  difference  between  the  three  devices  can  be 
performed.  An  analysis  of  variance  (ANOVA)  model  allows  incorporation  of  more 
independent  variables  into  the  model,  such  as  test  order  or  survey  responses. 
One  possible  ANOVA  model  in  a  study  such  as  this  one  is  given  in  Figure  1 1 . 


//J> 

-//  +  «,.+  /?.  +0k+  £ijk 

where: 

/ 

the  device  tested  {A,  B,  D} 

j 

the  participant  #  {0500. ..0599} 

k 

the  order  tested  {1  or  2} 

y,j* 

the  task  time  for  this  combination 

a, , 

the  device  effect  for  device  / 

Pj 

the  participant  effect  for  participant  j 

ok 

the  order  effect  for  order  k 

SUM 

random  error  □  n(/j,cj) 

Figure  11. 

General  ANOVA  Model  for  Usability  Testing 
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Using  ANOVA,  a  difference  was  not  detected  based  on  task  3  (read  grid 
coordinates  from  a  map): 

>  anova ( aov ( t3~dev , da ta=rpda ) ) 

Analysis  of  Variance  Table 

Df  Sum  of  Sq  Mean  Sq  F  Value  Pr(F) 

dev  2  29082  14541  2.14  |0 . 188| 

Residuals  7  47559  6794 

As  expected,  when  adjusting  for  participant  effect,  the  statistical 
significance  of  the  test  was  even  lower: 


>  anova (aov (t3adj~dev,data=rpda) ) 

Analysis  of  Variance  Table 

Df  Sum  of  Sq  Mean  Sq  F  Value  Pr(F) 


dev  2 
Residuals  7 


0.221  0.110 
0.452  0.065 


1.71  0 . 2  4  £ 


In  task  5  (the  unadjusted  time  to  send  a  message),  a  statistically 
significant  difference  was  found  (see  Figure  12),  and  it  was  initially  concluded 
that  using  the  device  RPDA-D  is  in  fact  faster  than  using  RPDA-A  (and  B  is  also 
better  than  RPDA-A  at  a  slightly  lower  confidence  threshold): 
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>  t5 . aov=aov (t5~dev, data=rpda) 

>  anova (t5 . aov) 

Analysis  of  Variance  Table 

Df  Sum  of  Sq  Mean  Sq  F  Value  Pr(F) 


Residuals  7  10689  1527 

>  multicomp (t5 . aov) 

95  %  simultaneous  confidence  intervals  for  specified 
linear  combinations,  by  the  Tukey  method 

Estimate  Std. Error  Lower  Bound  Upper  Bound 


A-B 

98.6 

35.7 

-6.47 

204 . 0 

A-D 

85.6 

28.5 

1.59 

170.0  **** 

B-D 

-13.0 

32.7 

-109.00 

83.3 

>  plot (multicomp (t5 . aov) ) 


A-B 

A-D 

B-D 


-120  -80  -40  0  20  40  60  80  120  160  200 

simultaneous  95  %  confidence  limits,  Tukey  method 
response  variable:  t5 

Figure  12.  ANOVA  analysis  of  task  5 
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However,  when  the  time  was  adjusted  by  the  individual’s  conventional 
task  time,  a  difference  was  not  seen.  Therefore,  the  better  task  times  may  not  be 
due  to  better  devices,  but  to  better  operators,  better  training,  or  some  other  effect 
(see  Figure  13). 


>  anova (aov (t5adj~dev,data=rpda) ) 

Analysis  of  Variance  Table 

Df  Sum  of  Sq  Mean  Sq  F  Value 

Pr  (F) 

dev  2 

91  45.3  1.81 

0.232 

Residuals  7 

175  25.0 

Figure  13. 

ANOVA  analysis  of  task  5,  adjusted  for  participant 
effect 

There  were  also  slight  differences  in  the  scenario  presented  during  the 
testing  of  RPDA-D  and  of  RPDA-A.  RPDA-B  used  a  very  abbreviated  scenario 
with  only  two  tasks.  Although  the  tasks  were  similar,  the  RPDA-B  scenario  was 
not  at  all  comparable  to  the  scenario  presented  during  the  testing  of  RPDA-A  and 
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RPDA-B.  This  additional  confound  complicated  statistical  analysis  to  the  point  of 
making  it  difficult  to  justify  with  such  small  sample  sizes. 

ANOVA  is  subject  to  assumptions  of  normality  and  of  equal  variances. 
Both  assumptions  are  highly  suspect  with  this  data.  First,  the  data  are  known  to 
not  be  normally  distributed.  Task  times  cannot  fall  below  zero,  yet  some  tasks 
have  short  task  time  and  high  variability.  The  tasks  are  also  seen  to  have  heavy 
right  tails,  with  occasional  users  taking  extremely  long  times. 

When  using  ANOVA  on  data  such  as  these,  variance-stabilizing 
transforms  may  be  considered  to  address  the  impact  of  unequal  variances. 
These  data  appeared  to  become  more  normally  distributed  with  both  logarithmic 
and  even  square-root  algorithms,  but  with  the  very  small  sample  sizes,  the 
appropriate  transform  is  unclear. 

To  ensure  against  unwarranted  assumptions,  a  two-sided  Kruskal-Wallis 
rank  sum  test  was  performed  on  the  same  data.  Rank-based  tests  are  especially 
appealing  because  of  the  heavy  tails  of  the  distributions.  Figure  14  demonstrates 
non-parametric  testing  of  task  3,  task  5,  and  task  5  adjusted  for  participant  effect. 


>  kruskal . test (t3 ,  dev) 

Kruskal-Wallis  chi-square  =  4.170,  df  =  2,  p-value  =  |0 . 12  4| 


>  kruskal . test (t5 ,  dev) 

Kruskal-Wallis  chi-square 


6.070,  df  =  2,  p-value 


0.048 


>  kruskal . test (t3adj ,  dev) 

Kruskal-Wallis  chi-square  =  3.386,  df  =  2,  p-value 


0.184 


Figure  14.  Kruskal-Wallis  tests  for  device  effects 


When  using  S-Plus,  Kruskal-Wallis  multiple  comparisons  are  performed 
manually.  Since  a  difference  was  detected,  the  multiple  comparisons  were 
performed  on  the  task  5  times,  at  a  95%  confidence  level  (Figure  15). 
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>  t5rank=rank (t5) ;  t5rank 

[1]  10.0  9.0  8.0  5.0  1.0  6.0 

>  dev 

[1]  AAABBDDDDD 

>  R=c (0,0,0) 

>  R [1] =  sum(t5rank [dev=="AM] ) ;R[1] 

[1]  27 

>  R [2 ] =  sum ( t5 rank [dev=="BM] ) ;R[2] 

[1]  6 

>  R [3] =  sum(t5rank [dev=="DM ] ) ;R[3] 

[1]  22 

>  n=c (length ( t5rank [dev=="A" ] ) ,  length ( t5rank [dev=="B" ] ) , 

length (t5 rank [dev=="D" ])) ;  n  #  n[i]  =  number  of  obs . 

[1]  3  2  5 

>  N=sum(n)  ;  N  #  N  =  total  obs. 

[1]  10 

>  k=  length (levels (dev) )  #  number  of  treatments 

>  alpha  =0.05  #  desired  confidence  level 

>  S . squared  =1/(N-1)  *  (sum (t5rankA2)  -  N*(N+1)A2  /  4) ; 

S .  squared 

[1]  9.111 

>  T.stat  =  (sum(RA2/n)  -  N*(N+1)A2  /  4)  /  S. squared;  T.stat 

[1]  6.069 

>  student . t . stat  =  qt (l-alpha/2 ,  N-k) ;  student . t . stat 

[1]  2.36 

>  cons tan t=student . t . stat  *  sqrt (S . squared  *  (N-l- 

T .  stat) / (N-k) ) ;  constant 

[1]  4.62 

The  following  lines  display  the  left-hand  side  and  RHS  of  the 
multiple  comparison  criteria  for  the  Kruskal-Wallis  test.  If  the  LHS  > 
RHS,  then  there  is  a  difference. 


>  i=l ;  j=2 ;  c( 

abs  (R  [  i  ] 

/ 

n  [i] 

-  R[j] 

/ 

n[j]) 

,  constant  * 

sqrt (l/n [i]  + 

l/n[ j] )  ) 

[1]  6.00  4.22 

>  i=l;  j=3 ;  c( 

abs  (R[i] 

/ 

n  [i] 

-  R[j] 

/ 

n  [  j  ] ) 

,  constant  * 

sqrt (l/n [i]  + 

l/n[ j] )  ) 

[1]  4.60  3.37 

>  i=2 ;  j=3 ;  c( 

abs  (R[i] 

/ 

n  [i] 

-  R[j] 

/ 

n  [  j  ] ) 

,  constant  * 

sqrt (l/n [i]  +  l/n[j])  ) 

[1]  1.40  3.86 

Figure  15.  S-Plus  Code  for  Kruskal-Wallis  Multiple  Comparisons 

Tests 


#  compute  the  ranks 

2.0  3.5  7.0  3.5 


#  initialize  array 

#  compute  the  rank  sums 
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At  a  =  0.05,  B  is  better  than  A,  and  D  is  better  than  A.  However,  B  is  no 
better  or  worse  than  D.  As  with  the  ANOVA  test,  however,  if  the  heuristic 
adjustment  for  participant  effect  on  task  5  is  performed,  a  statistically  significant 
difference  is  not  seen: 

>  kruskal . test (t5adj ,  dev) 


Kruskal-Wallis  chi-square  =  3.68,  df  =  2,  p-value  =  0.159 


4.  Inter-Rater  Reliability 

To  verify  that  the  measurements  are  objectively  coded,  inter-rater 
reliability  (IRR)  analysis  was  performed.  Task  time  assessments  by  two  different 
researchers  (“A”  and  “B”)  on  the  same  video  file  (participant  0513  operating 
device  RPDA-D)  are  shown  in  the  "confusion  matrix"  in  Table  19. 


Table  19. 

Confusion  Matrix  for  Participant 

0513,  RPDA-D 

tl 

t2 

t3 

t4 

t5 

t6 

t7 

Error 

Total  A 

tl 

22.63 

- 

- 

- 

- 

- 

- 

0.33 

22.96 

t2 

- 

54.73 

- 

- 

- 

- 

- 

0 

54.73 

t3 

- 

- 

9.71 

- 

- 

- 

- 

0 

9.71 

t4 

- 

- 

- 

25.1 

- 

- 

- 

0.14 

25.24 

t5 

- 

- 

- 

- 

92.06 

- 

- 

0 

92.06 

t6 

- 

- 

- 

- 

- 

19.53 

- 

5.8 

25.33 

t7 

- 

- 

- 

- 

- 

- 

5.5 

0.94 

6.44 

Error 

0 

1.63 

1.65 

0 

0.17 

0 

0 

- 

3.45 

Tot.B 

22.63 

56.36 

11.36 

25.1 

92.23 

19.53 

5.5 

7.21 

239.92 

Table  19  shows  the  total  duration  (in  seconds)  scored  for  the  seven  tasks 
(tl  -  t7)  by  two  different  coders  (“A”  and  “B”).  The  actual  time  coded  by  coder  A 
is  shown  in  the  Total  A  column  at  the  right,  and  the  actual  time  coded  by  coder  B 
is  shown  in  the  Total  B  row  at  the  bottom.  The  number  in  the  matrix  reflects  the 
agreement  between  the  two  times,  that  is,  the  lesser  of  the  two  times.  For 
example,  both  coders  agreed  that  the  participant  spent  at  least  22.63  seconds  on 
task  1.  Coder  A  actually  measured  the  time  as  22.96  seconds.  The  0.33  seconds 
of  difference  is  shown  in  the  Error  column  for  A.  Other  types  of  inter-rater 
reliability  analysis  could  have  additional  times  in  the  off-diagonal  cells  of  the 
confusion  matrix,  showing  what  task  or  behavior  the  coder  recorded  during  the 
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times  of  disagreement.  The  frequency  and  sequence  of  observed  behaviors  can 
also  be  compared. 

Figure  16  shows  that  the  task  times  (t1-t7)  assigned  by  different 
researchers  (on  the  vertical  and  horizontal  axes)  are  in  very  good  agreement  for 
this  observation. 


Figure  16.  Tasks  times  as  coded  by  Rater  A  and  Rater  B 


Statistics  commonly  used  to  ensure  that  objective  criteria  are  being 
applied  in  scoring  the  behaviors  include  the  index  of  concordance  and  Cohen’s  k. 
The  index  of  concordance  is  the  proportion  of  agreements  to  total  observed  time. 
Cohen's  k  is  a  measure  of  the  proportion  of  agreement  that  exceeds  that 
expected  by  chance  (Figure  17). 


K=P^ 

l~Pc 

where:  p0  =  the  observed  proportion  of  agreements 

pc  =  the  proportion  of  agreements  expected  by  chance 

Figure  17.  Formula  for  Cohen’s  k 


In  this  comparison  of  raters  A  and  B,  the  total  duration  of  agreements  is 
229.3  seconds,  the  total  duration  of  disagreements  is  10.66  sec,  the  index  of 
concordance  is  0.96,  and  Cohen’s  k  =  0.94.  This  indicates  that  independent 
observers  using  the  same  instructions  were  quantitatively  coding  behavioral  data 

in  a  very  reliable  manner.  Achieving  this  level  of  agreement  requires  detailed  and 
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objective  coding  criteria  and  scenarios  that  provide  for  precisely  defined  task 
start  and  end  points. 


D.  DISCUSSION  OF  CASE  STUDY 

1.  Conclusions  from  Case  Study 

Due  to  the  high  rate  of  participant  attrition,  statistically  supportable 
conclusions  cannot  be  drawn  from  the  quantitative  data  collected  in  this  study.  A 
number  of  usability  issues  were  directly  observed  or  were  noted  in  the  post¬ 
operation  surveys. 

a.  All  RPDAs  interfered  with  the  user’s  ability  to  use  their  personal 
weapon.  (HDE  0501AB,  0502A,  0505AB,  0506A,  0515D, 
0528D,  RA0502  at  20:14;  RA0503  at  28:15)4 

b.  Participants  were  generally  not  particularly  satisfied  or 
dissatisfied  with  a  stylus  input  device,  although  some  were 
highly  opposed.  Almost  all  participants  who  had  negative 
comments  about  the  stylus  had  never  used  one  before.  (HDE 
0501 AB,  0502B,  0505AB,  0506A  and  corresponding  TESs) 

c.  User  satisfaction  was  not  tied  to  objective  performance.  For 
example,  participant  0502  was  "Very  Dissatisfied"  with  the 
RPDA-B  stylus,  and  "Very  Satisfied"  with  RPDA-A.  However,  he 
was  actually  faster  in  the  messaging  tasks  with  RPDA-B  than 
with  RPDA-A.  The  video  shows  him  repeatedly  tapping  with  the 
stylus  using  RPDA-B,  which  is  a  source  of  frustration  but  was 
not  detrimental  to  his  overall  performance. 

d.  A  single  incident  can  disproportionately  affect  user  satisfaction 
with  the  overall  device.  Participant  0501  experienced  a  single 
incidence  of  difficulty  with  the  stylus,  but  no  other  problems.  His 
satisfaction  with  the  stylus  was  "Very  Dissatisfied",  although  his 
performance  using  it  was  average. 

e.  RPDAs  can  bring  inexperienced  personnel,  who  performed 
poorly  on  baseline  tasks,  up  to  performance  levels  similar  to 


4  The  parenthetical  comments  indicate  the  survey  questions  or  the  observations  that  support 
the  conclusion.  Surveys  are  prefaced  "HDE"  for  the  Handheld  Device  Evaluation  or  "TES"  for  the 
Technology  Experience  Survey,  followed  by  the  four-digit  participant  number.  RPDA  observations 
begin  with  "R",  followed  by  the  letter  identifying  the  RPDA,  followed  by  the  four-digit  participant 
number,  followed  by  the  approximate  time  (minutes:seconds)  in  the  video  that  the  behavior  was 
observed. 
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personnel  who  excel  at  baseline  tasks.  Participants  with  little 
military  experience  ("novices")  required  considerable  training  to 
perform  the  conventional  mapping  tasks  and  were  generally  two 
to  ten  times  slower  than  users  with  high  military  experience 
("experts").  The  novices  were  also  more  likely  to  make  errors  in 
conventional  tasks  than  the  experts.  However,  novice 
performance  using  the  RPDAs  for  mapping  was  comparable  to 
expert  performance.  Although  experts  generally  fared  better  at 
RPDA  mapping  than  novices,  they  were  only  incrementally 
better.  The  novices  cut  their  mapping  skill  task  time  by  a  factor 
of  ten  in  some  cases,  while  some  experts  actually  experienced 
an  increase  in  their  task  time  compared  to  conventional  means. 
(See  task  3,  participants  0502,  0503,  and  0527). 

f.  Users  commented  about  the  bulk  of  RPDA-B,  which  was  the 
largest  one-piece  unit,  but  expressed  even  stronger 
dissatisfaction  with  the  RPDA-A,  which  had  a  smaller  handheld 
piece  tethered  to  equipment  in  the  pack  (HDE  0501 A).  Several 
problems  (one  fairly  serious)  were  observed  with  the  tethered 
cord  become  tangled  in  other  equipment  (RA0503  at  28:15). 

g.  Lacking  feedback  and  appropriate  error  messages,  users 
wasted  much  time  attempting  incorrect  procedures.  Virtually  all 
participants  indicated  that  if  they  experienced  a  problem  using  a 
computer,  they  would  "keep  trying"  as  their  first  step  to  solve  it. 
This  behavior  was  frequently  observed  (RA0501A  at  37:50) 
when  the  device  did  not  give  feedback  to  the  user  that  an 
inappropriate  action  (such  as  scrolling  off  of  the  edge  of  a  map) 
was  being  attempted. 

h.  The  RPDAs  made  novice  users  feel  more  confident  in  their 
mapping  skill,  even  when  their  performance  and  accuracy  were 
poor.  (HDE  0502A,  0505A).  This  may  be  an  example  of  the 
false  pedigree  of  credibility  that  high-technology  devices  can 
carry,  as  discussed  in  the  literature  review  (page  11). 

i.  The  low  visibility  and  small  size  of  the  PDA  screen  (72  dots  per 
inch,  low-contrast  screen,  compared  to  a  1200  dots  per  inch, 
high-contrast  paper  map)  made  it  very  difficult  for  users  to 
search  for  locations  with  which  they  were  unfamiliar. 

2.  Recommendations  from  Case  Study 

Participants  in  this  study  had  only  an  introductory  level  of  proficiency  and 
experience  with  the  devices.  Performance  at  this  stage  should  not  be  of  primary 
interest,  except  to  the  extent  that  these  initial  impressions  make  an  impact  on 
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user  acceptance.  Users  proficient  with  the  stylus  and  the  software  should  be 
participants  in  testing  usability  and  user  satisfaction.  More  proficient  users  also 
have  reduced  intra-task  variability  and  would  provide  better  data  for  statistical 
analysis. 

The  recommendations  for  context-aware  designs  found  in  the  introduction 
to  this  thesis  (page17)  have  been  confirmed  by  the  qualitative  results  of  this 
study: 

Sensible  default  behavior.  Users  benefited  from  RPDAs  that  were 
organized  consistently  and  performed  appropriate  actions  the  first  time.  An 
example  of  good  default  behavior:  tapping  a  location  on  a  map  brings  up  the 
most  likely  information  desired  (grid  location).  An  example  of  poor  default 
behavior:  when  reading  a  message,  RPDA-D  required  users  to  scroll  to  the 
bottom  of  the  message  (past  header  information  that  was  never  used)  to  get  the 
information  they  want  -  the  message  text.  The  header  information  should  be  less 
prominent  by  default. 

Restricted  options.  Errors  were  frequently  seen  when  users  are  allowed 
to  perform  incorrect  tasks,  such  as  closing  a  necessary  application,  or  changing 
vital  communications  settings.  Users  should  be  protected  from  actions  that  would 
be  incorrect  in  most  situations. 

Robustness  to  inputs.  Users  became  extremely  frustrated  when  simple 
slips  caused  serious  errors.  Slips  appeared  to  occur  due  to  users  having  to  split 
their  attention  or  due  to  input  errors  with  the  stylus. 

Informative  error  conditions  and  graceful  recovery.  RPDAs  should 
provide  immediate  feedback  to  the  user  when  inappropriate  tasks  are  attempted, 
or  they  will  likely  continue  to  attempt  that  task  incorrectly  before  trying  other 
solutions.  This  behavior  was  reported  on  almost  all  the  surveys  and  directly 
observed  in  many  instances. 

The  appropriate  level  of  detail  presented  on  a  map  appears  to  be 
extremely  important.  Displaying  multiple  scales  of  map  (for  example,  1:250,000 
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for  zoomed  out  and  1:50,000  for  zoomed  in)  may  help  alleviate  some  of  the 
confusion  users  experienced  when  zoomed  out  on  a  cluttered,  highly  detailed 
map. 

Menus  and  action  buttons  should  be  consistent  and  readily  visible.  Users 
were  frequently  observed  looking  in  the  wrong  menus  for  functions  that  appeared 
in  that  location  in  another  application.  Users  do  not  generally  want  (or  need)  to 
understand  that  different  applications  may  be  running  simultaneously  (such  as 
separate  communications  and  navigation  applications).  Different  applications 
should  still  maintain  similar  interfaces.  For  example,  when  changing  between 
communication  and  navigation  functions  on  RPDA-A,  the  menu  selections 
change  subtly.  In  the  communications  mode,  users  consistently  went  to  the 
incorrect  menu  (which  would  have  been  the  correct  menu  in  the  navigation 
application)  to  change  applications. 

3.  Follow-on  Research  to  the  Case  Study 

The  framework  of  the  study  remains  in  place,  including  points  of  contact, 
method,  and  equipment.  Adding  additional  data  points  with  any  of  the  devices,  in 
order  to  allow  more  robust  statistical  analysis,  should  be  a  relatively 
straightforward  process.  Continued  work  with  the  RPDA  case  study  may  yield 
definitive  results,  allowing  comparison  between  RPDAs,  and  additional 
recommendations  for  the  field  of  RPDA  design.  Integrating  the  Applied 
Warfighter  Ergonomics  Center’s  laboratory  and  field  capabilities  early  on  in  the 
design  and  prototyping  of  RPDAs  and  other  military  individual  equipment  will 
result  in  better  equipment  being  fielded.  The  AWE  laboratory  could  be  used  in 
the  design  of  RPDAs  in  the  following  ways: 

•  defining  the  appropriate  tasks  to  be  performed  by  an  RPDA, 

•  quantifying  the  effect  of  RPDA  use  on  battlefield  situational 
awareness  (SA), 

•  defining  the  appropriate  characteristics  of  an  RPDA:  device 
size,  weight,  screen  size,  reflectivity,  brightness,  resolution, 
and  types  of  controls. 
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A  number  of  questions  surfaced  during  the  planning,  execution,  and 
analysis  of  this  case  study  that  are  of  keen  interest  but  beyond  the  scope  of  this 
study.  The  following  topics  would  be  excellent  avenues  to  pursue  a  more  detailed 
study  using  the  capabilities  of  the  Applied  Warfighter  Ergonomics  Observation 
Laboratory. 

Technology  experience  among  various  U.S.  military  populations.  The 

Technology  Experience  Survey  is  ready  to  be  issued  in  a  full  study,  and  concise 
enough  to  reasonably  administer  to  large  populations  without  excessive 
coordination  with  the  participating  units.  Some  military  surveys  are  so  extensive 
as  to  be  very  demanding  on  operating  forces  with  precious  little  time  to  spend 
supporting  such  external  demands.  Although  web-based  surveys  are  currently 
popular  due  to  the  ease  with  which  they  can  be  administered  and  scored,  this 
survey  is  a  particularly  poor  choice  to  be  administered  via  electronic  means.  A 
large  canvassing  of  different  populations  could  be  used  in  several  ways.  First, 
natural  groupings  of  responses  could  be  used  to  classify  persons  of  high, 
moderate,  and  low  technology  experience,  and  such  ratings  could  be  used  as 
independent  variables  in  future  studies.  The  population  sizes  in  the  case  study 
were  too  low  to  adequately  break  down  groups  in  this  manner.  Then,  the  actual 
technology  experience  of  groups  of  interest  to  military  technology  developers 
could  be  quantified.  Rather  than  assuming  the  likely  users  are  of  the  “Nintendo 
generation,”  a  device  targeted  at  Army  Ranger  team  leaders  could  be  designed 
from  the  start  with  actual  data  about  the  technology  experience  of  those  users. 

Baseline  performance  metrics  of  standard  tactical  tasks.  The  tactical 
tasks  used  in  this  study,  or  other  relevant  tasks,  could  be  measured  in  larger 
populations  to  establish  baseline  performance  metrics  such  as  task  time  and 
error  rate. 

Performance  differences  between  populations  of  interest.  In  this 

study,  students  at  the  Defense  Language  Institute  were  used  as  surrogates  for 

the  real  users  of  interest.  Naval  Postgraduate  School  students  were  not  used 

under  the  untested  assumption  that  their  technology  experience  greatly 
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surpasses  that  of  common  military  users.  This  laboratory  could  be  used  to 
definitively  establish  the  performance  differences  between  populations  such  as 
typical  Marine  infantrymen,  DLI  students,  and  civilian  software  programmers. 
This  information  could  be  used  by  developers,  testers,  and  program  managers  to 
establish  (or  question)  the  external  validity  of  studies  that  use  test  personnel  from 
outside  the  actual  populations  receiving  the  equipment. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

With  the  Applied  Warfighter  Ergonomics  Center,  the  Department  of 
Defense  now  has  an  independent,  organic  capability  to  perform  individual 
equipment  usability  testing,  rather  than  relying  on  defense  contractors  and  ad 
hoc  testing.  Military  individual  equipment  can  be  scientifically  evaluated  under 
controlled  conditions  in  the  laboratory  and  realistic  conditions  in  the  field 

Innovations,  such  as  RPDAs,  that  demonstrably  improve  productivity,  are 
useless  if  they  are  not  adopted  by  warfighters  in  the  field.  More  extensive 
research  may  find  tradeoffs  or  reinforcement  between  the  factors  that  motivate 
adoption  and  the  factors  that  drive  usability.  Increasing  user  satisfaction,  even  in 
ways  that  limit  the  functionality  of  the  equipment,  may  result  in  more  widespread 
adoption  and  ultimately  a  better  fighting  force.  Feature-laden  devices  that 
frustrate  or  confuse  users,  and  are  not  adopted,  will  be  just  dead  weight  in  the 
bottom  of  a  pack. 

This  research  introduces  a  validated,  generalizable  way  of  measuring 
what  equipment  works  best  for  specific  populations  of  users,  and  what  equipment 
those  users  are  most  satisfied  with.  Both  equipment  and  training  can  be 
designed  so  that  functionality  is  accessible  to  the  actual  users  (not  just  to  the 
engineer  on  the  next  bench),  and  so  that  users  will  be  inclined  to  fully  utilize  the 
equipment  of  their  own  volition. 

B.  RECOMMENDATIONS 

The  Applied  Warfighter  Ergonomics  Observation  Laboratory  is  primarily 
equipped  for  collecting  measurements  and  establishing  quantitative  comparisons 
on  individual  military  equipment.  Nothing  in  the  laboratory  inherently  lends  itself 
directly  to  creating  better  designs,  although  it  can  (and  should)  be  integrated  into 
the  early  design,  prototyping,  and  subsequent  testing  of  many  different  types  of 
equipment.  Early  user  assessment,  continuous  testing  and  feedback,  and  in- 
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progress  validation  are  key  capabilities  that  this  laboratory  enables  for  military 
individual  equipment.  Appropriate  program  offices  should  be  made  aware  of  the 
laboratory’s  capabilities  so  that  ultimately,  individual  soldiers,  sailors,  and 
Marines  can  benefit  at  a  personal  level  from  improved  equipment  made  possible 
by  the  AWE  Center. 

The  equipment  in  the  laboratory  is  complicated  and  experienced  users  are 
required  for  the  laboratory’s  continued  operation  and  maintenance.  A  concerted 
effort  will  be  required  to  maintain  continuity  in  student  skills  with  the  laboratory 
equipment.  Incorporating  various  parts  of  the  lab’s  capabilities  into  existing  and 
future  courses  will  stimulate  interest  and  increase  the  pool  of  capable  users. 

As  shown  by  Carpenter  and  Nakamoto  (1994),  the  behavior  of  novice 
users  and  early  adopters  may  not  be  indicative  of  mainstream  behavior.  Users 
will  tend  to  adapt  their  preferences  towards  the  technology  as  it  becomes 
adopted.  So,  the  preferences  of  the  first  users  may  not  be  a  good  gauge  of  either 
qualitative  satisfaction  or  quantitative  performance.  Task  times  and  error  rates 
observed  early  in  Jordan's  (1998)  learning  curve  are  of  less  interest  than 
measurements  taken  of  more  experienced  users.  More  experienced  users  will  be 
more  representative  of  how  the  equipment  will  actually  be  used  in  combat.  A 
concerted  effort,  involving  command  interest  at  the  highest  possible  levels,  must 
be  made  to  commit  appropriately  representative  users  to  usability  testing  in  this 
laboratory. 

C.  FOLLOW-ON  RESEARCH 

The  field  capabilities  of  the  laboratory  have  been  tested  in  small  studies  at 
Camp  Roberts  and  in  mobile  data  collection  on  the  NPS  campus.  A  full  exercise 
of  the  behavioral  observation  capabilities  in  an  actual  deployed  environment  has 
not  been  performed.  A  full  demonstration  of  this  deployable  capability,  to  validate 
the  results  of  a  laboratory  study,  would  complete  the  establishment  of  this 
laboratory  as  a  premiere  center  for  research  in  military  individual  equipment. 
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The  AWE  lab  is  also  potentially  a  source  of  justifiable  human  performance 
data  for  computer  models  and  combat  simulations,  providing  not  just  point 
estimates,  but  empirically  derived  distributions.  The  lab  can  be  used  to  accurately 
measure  task  times,  error  rates,  and  training  effectiveness  in  a  wide  variety  of 
military  applications. 
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APPENDIX  A.  GLOSSARY  AND  ACRONYMS 


Some  terms  have  a  unique  meaning  within  a  specific  context,  such  as  within  the 
Noldus  Observer  software.  In  those  cases,  the  context  is  indicated  in  [brackets]. 


1:50,000 

The  scale  of  standard  military  maps  used 
by  ground  forces.  Such  a  map  might  be 
called  a  “one  to  fifty”. 

AWE 

Applied  Warfighter  Ergonomics 

BDUs 

Battle  dress  uniforms  worn  by  United 
States  Army  personnel. 

Bearing 

Direction  from  one  point  on  a  map  to 
another  point.  Bearing  may  be  given  in 
relation  to  magnetic  north  (as  read  on  a 
compass)  or  true  north  (as  read  from  a 
map). 

Events  that  are  recorded  during  a  scenario 
are  called  behaviors.  There  are  event 
behaviors,  which  occur  at  a  single  instant 
in  time,  and  state  behaviors,  which  have  a 
duration.  Behaviors  are  grouped  into 
Behavioral  Classes. 

Cammies  Camouflage  utility  uniforms  worn  by 

United  States  Marines. 

CDR  Combat  Decision  Range 

Communications  Check  (Comm  An  initial  exchange  used  to  verify  that 

Check)  communications  gear  is  functioning.  Users 

on  each  end  will  send  and  respond  with  a 
message. 


Behaviors  [Noldus  Observer 
Software] 
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Configuration  [Noldus  Observer  A  configuration  should  not  be  changed 
Software]  after  starting  to  collect  data.  A 

configuration  is  locked  after  data  files  are 
made.  If  you  unlock  the  configuration,  you 
must  remove  the  data  files,  make  changes 
to  the  configuration,  and  then  add  the  data 
files  back. 

DV  Dependent  variable.  Measurable  data  that 

can  be  predicted  by  the  independent 
variables  (IVs). 


Grid,  Grid  Coordinate  The  standard  ground  combat  military 

method  to  identify  points  on  a  map  (as 
opposed  to  latitude  and  longitude,  used  in 
aircraft  and  ship  navigation).  A  four-digit 
grid  coordinate  (or  just  “grid”)  is  accurate 
to  within  1,000  meters.  A  six-digit  grid  is 
accurate  to  within  100  meters.  An  eight¬ 
digit  grid  is  accurate  to  within  10  meters, 
and  is  the  practical  limit  of  accuracy  in 
most  current  navigation  devices,  such  as 
reading  a  1:50,000  scale  military 
topographical  map. 

HCI  Human-Computer  Interaction 

HF  Human  Factors 

HSI  Human  Systems  Integration 

IV  Independent  Variables  (IVs)  represent 

specific  information  about  the  participants 
and  the  scenario  in  any  particular 
observation  session.. 


IRB  Institutional  Review  Board 

ISO  International  Organization  for  Standards 

(acronym  in  French) 
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Leg 


A  single  path  from  one  point  on  a  map  to 
another  point,  often  part  of  a  route 


MEF 

Military  individual  equipment 

ORD 

PDA 

Patrol  Route 


Marine  Expeditionary  Force 

Equipment  that  is  normally  carried  and 
employed  by  a  single  person  in  combat 

Operational  Requirements  Document 

Personal  Digital  Assistant  (handheld 
computer) 

see  Route 


Projects  [Noldus  Observer 
Software] 


Range 


Route 


RPDA 


Scoring  or  Coding 


A  project  file  contains  the  configuration,  all 
observation  and  data  files,  and  analysis  for 
a  given  experiment.  You  can  only  do 
analysis  on  data  within  the  same  project. 

Straight-line  distance  from  one  point  on  a 
map  to  another  point. 

A  series  of  connected  straight  lines  (legs) 
used  to  navigate  from  one  point  to  a  final 
destination,  usually  through  several 
intermediate  points. 

Ruggedized  Personal  Digital  Assistant.  A 
handheld  computer  that  is  designed  for 
outdoor  use  in  a  variety  of  environments. 
RPDAs  are  commonly  based  on 
conventional  Personal  Digital  Assistants 
(PDAs)  running  Palm  OS,  Microsoft 
Windows  Mobile  Edition  (also  called 
Windows  CE  or  PocketPC),  standard 
Microsoft  Windows,  or  Linux. 

The  process  of  turning  observed  behaviors 
into  a  data  file  for  later  analysis. 
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Subjects  [Noldus  Observer 
Software] 


TNT 


Ul 

VOIP 


Workspace  [Noldus  Observer 
Software] 


The  roles  (“billets”  or  "actors")  played  by 
the  people  simultaneously  participating  in 
an  observation  session,  such  as  “Team 
Leader”  and  “Rifleman”.  Very  often  there 
will  be  only  a  single  subject  (the  person 
using  the  equipment  being  tested),  and  so 
subjects  will  not  be  set  up  in  Observer. 

Tactical  Network  Topology.  An  ongoing 
NPS  research  project,  formerly  known  as 
"STAN" 

User  Interface 

Voice  over  Internet  Protocol.  A  means  to 
send  digitized  audio  over  networks  using 
Internet  Protocol  (IP).  VOIP  has  the 
advantage  of  using  common  networks,  but 
is  inefficient  compared  to  other  ways  of 
sending  digital  audio. 

An  Observer  workspace  is  a  way  to 
organize  what  projects  are  visible.  Use 
different  workspaces  for  different 
researchers,  or  completely  different 
research.  The  Observer  system  in  the 
AWE  center  has  a  Sample  Workspace,  a 
workspace  for  Class  Projects,  and 
separate  workspaces  for  theses  and  other 
research. 
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USABILITY  LAB  BASIC  USER  GUIDE 


Applied  Warfighter  Ergonomics 
Observation  Laboratory 
Basic  User  Guide 


[WPS 


85 


First  published:  August,  2005 

Koichi  Takagi 
Captain,  USMC 

Naval  Postgraduate  School 
Monterey,  California 


86 


Table  of  Contents _ 

Introduction .  1 

Capabilities . 1 

Examples  of  possible  results . 2 

How  to  use  this  manual . 3 

The  basics  of  observational  research . 3 

Preparation  before  an  observation  session . 5 

Recording  a  Session  (Laboratory) . 7 

Recording  a  Session  (Field) . 10 

Scoring  a  Session . 1 1 

Definitions . 1 1 

Noldus  Observer  step-by-step  tutorial . 13 

Analyzing  Data . 21 

Exporting  data  from  The  Observer . 21 

Using  The  Observer's  analysis  tools . 21 

Video  Equipment:  Other  Capabilities . 27 

Capturing  a  VHS  video  tape  to  a  computer  file . 27 

Burning  video  files  to  DVD . 28 

Appendix  A.  Observer:  Advanced  Capabilities . 29 

Appendix  B.  Observer  Lab  Components . 31 

Observer  station  components . 31 

Observer  workstation  (special  hardware  and  software) . 31 

Participant  station  components . 32 

Appendix  C.  Observation  Lab  Technical  Reference . 33 

Observer  Station . 33 

Participant  station . 41 

System  Hardware  Components . 43 

Wiring  Diagrams . 44 

Screen  Shots  of  Typical  Configurations . 47 

Appendix  D.  Portable  Observation  Laboratory  Embarkation  List . 48 

Appendix  E:  Useful  signs  and  displays . 52 

References . 55 


87 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


88 


Introduction 

The  Naval  Postgraduate  School  (NPS) 

Applied  Warfighter  Ergonomics  {AWE) 

Center's  Observation  Laboratory  provides  a 
laboratory  and  established  methodologies 
for  scientifically  evaluating  the  human 
factors  considerations  of  combat  equipment 
used  by  military  personnel  under  both 
controlled  and  realistic  field  conditions. 

This  User's  Guide  provides  detailed,  step- 
by-step  instructions  and  a  tutorial  for  using  the  laboratory  in  a  basic  configuration 
suitable  for  learning  the  system  and  for  conducting  a  small  study. 


Capabilities 

Technical  capabilities  of  the  Observation  Laboratory 
include: 

Capturing  behavior  in  the  laboratory  using  digital 
remote  video  cameras  and  fixed  and  wireless 
microphones. 

Recording  professional  quality  digital  video  in  a  laboratory  or  field  environment  to 
computer  flies  and  DVDs,  using  portable  miniDV  camcorders,  or  mobile  device 
camera  with  portable  miniDV  VCR. 

Mixing  and  recording  audio  and  video  from 
up  to  four  sources  (such  as  video  cameras  or 
video  tape)  onto  AVI  i  MPEG-2  computer 
files,  DVD,  or  videotape. 

Producing  highlight  videos  of  events  of 
interest. 

Editing,  producing,  and  analyzing 
observations  using  Noldus  Observer 
software  to  score  observation  sessions  and  perform  statistical  analysis. 

Direct  screen -capture  of  the  participant's  computer  screen  to  the  observer's 
screen  and  to  video. 

Coding  behavior  and  other  events  to  time-stamped  logs  that  are  tied  to  the  video 
files.  Directly  perform  statistical  analysis  on  the  logs  and  jump  to  coded  events  on 
video. 
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Research  capabilities  include: 

Observe  participant(s)  while  using  a  computer  or  other  equipment  in  the  lab. 
Record  comments  during  an  after-action  report  (AAR)  de-brief. 

Record  interviews  and  focus  group  studies. 

Record  participants  reactions  to  projected  video,  or  to  scenarios  read  over 
loudspeaker. 

Perform  reliability  analysis  between  coders. 

Perform  lag-sequential  analysis  on  behaviors  and  events. 


Examples  of  possible  results 

"When  given  the  task,  'locate  yourself  on  the  [computer]  map',  7  out  of  1 6 
participants  (44%)  performed  it  correctly  within  the  allotted  60  seconds  (mean 
time  to  complete  =  45.7s,  u=  9.8s),  5  participants  (31%)  completed  the  task  after 
making  one  or  more  errors  (p  =  88.3  s,  u=  15.2  s),  and  4  participants  (25%)  did 
not  complete  the  task  within  5  minutes  (gave  up  or  experienced  a  technical 
malfunction)." 


"Buttons  1  and  2  were  pressed  correctly  on  the  first  try  98%  of  the  time,  button  3 
was  pressed  correctly  on  the  first  try  87%  of  the  time,  but  button  4  was  pressed 
correctly  on  the  first  try  only  45%  of  the  time.  Errors  on  button  4  included  missing 
the  button  (28%  of  the  time),  pressing  the  wrong  button  (18%  of  the  time),  and 
dropping  the  device  (9%  of  the  time)." 


"During  the  After  Action  Report  (AAR)  de-brief,  team  member  #3  appeared  to 
scowl  or  otherwise  disagree  with  the  team  leader's  comments  on  6  occasions, 
but  made  no  verbal  comment.  Members  1 , 2,  and  6  consistently  reinforced  each 
other's  comments  verbally  and  non-verbally." 


"The  equipment  load  as  worn  interfered  with  the  participants'  ability  to  shoulder 
their  weapons.  It  increased  the  time  to  move  from  a  standing  position  to  a 
covered,  kneeling  firing  position  by  an  average  of  3.2  seconds  (n=  21  subjects, 
average  of  5  trials  per  subject,  p=0.043).  Here  are  some  example  video  clips  of 
the  movement  with  and  without  the  carrying  load,  showing  how  it  interfered." 
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How  to  use  this  manual 

This  manual  is  designed  to  allow  a  beginning  student  researcher  to  perform  a 
basic  usability  analysis  in  the  AWE  Usability  Lab.  The  equipment  and  procedures 
can  be  technically  challenging  at  first,  although  you  should  be  able  to  fully 
complete  the  tutorial  (including  recording  and  analyzing  a  test  case)  within  three 
hours.  It  is  imperative  that  you  run  through  at  least  one  test  recording  before 
trying  to  conduct  even  a  small  research  study. 


1 .  Before  designing  your  experiment,  read  The  basics  of  observational  research. 

2.  When  ready  to  conduct  your  usability  testing,  follow  the  step-by-step  tutorials 
in  Preparation  before  an  observation  session  and  Recording  a  session 
(Laboratory).  In  these  steps,  you  will  create  video  files  of  your  participants.  You 
should  also  refer  to  Appendix  C:  Observation  Lab  Technical  Reference. 
Appendix  C  details  how  to  avoid  or  recover  from  the  common  problems  you  will 
encounter. 

3.  To  analyze  your  video  files,  follow  the  step-by-step  tutorials  in  Scoring  a 
Session  and  Analyzing  Data. 


The  other  sections  of  this  manual  will  be  useful  in  configuring  and  upgrading  the 
lab,  and  in  guiding  you  in  pursuit  of  more  sophisticated  use  of  the  facility 


The  basics  of  observational  research 

To  use  the  AWE  Center  Observation  Lab  effectively,  researchers  should  prepare 
the  following: 

1 .  A  well-formed  research  question  to  investigate. 

2.  A  list  of  the  independent  variables  (I  Vs)  that  will  be  controlled  in  each 
observation  session.  This  may  include  information  about  the  participant  (name, 
rank,  gender,  etc.)  and  information  about  the  scenario  or  environment  ("wearing 
equipment  load  A",  "night  conditions",  etc.) 

3.  A  list  of  tasks  to  be  given  to  the  participant  (e.g.  "send  a  message") 

4.  A  detailed  list  of  "behaviors"  (or  "events")  that  will  be  observed  and  recorded. 
Virtually  all  possible  behaviors  of  interest  should  be  defined  before  hand  ("start 
task",  "press  button  T' "user  commits  error"...,  "complete  task")  Not  all  events  will 
necessarily  occur,  and  some  may  occur  multiple  times. 

5.  A  detailed,  scripted  scenario,  including: 

*  A  timeline. 

•  A  description  of  what  each  researcher  and  participant  will  be  doing  at  each 
point  in  the  scenario. 
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*  irtdiMtiarts  6f  what  will  trigger  each  ev&ntor  behavior  of  interest  (a  3  how 

does  she  participant  knew  to  sla«a  tasit?). 

*  What  equipment  will  be  used  (i.e  helmet,  Fine,  handheld  device,  etc.). 

6.  Any  required  paperwork  such  a*  IRBs,  consent  farms,  background 
questionnaires,  etc. 


Recommended  introductions  to  usability  testing  include: 


Rubin,  J.  (1994).  Handbook  o/usaMffr  testing;  Hqw  to  plan, 
design,  and  conduct  eflfcd'jVe  teste.  New  Yorik:  Wiley. 

Jordan,  P.W.  ( 1 99-B)  An  introduction  to  usability.  Bristol,  PA 
Taylor  &  Francis.  Inc. 

Peftman.  G.  ( 2005)  HCt  bibliography;  Human-computer  interaction 
/ us&r  interface  usability.  Retrieved  August  2005,  2005  from 
h  ttp ;  //www .  he  i  b  i  b.  □  rq/ 


Researchers  will  produce  video  files  of  the  observation  session  for  each 
participant,  and  then  use  N  nidus  Observer  to  score  the  behaviors  observed.  The 
data  can  be  exported  as  Excel  files,  and  analyzed  however  you  please 

Different  researchers  can  scoie  the  same  sessions  from  video  files,  and 
aulomaliCBlIy  da  reliability  analysis  between  the  coders 

The  main  phases ot  using  the  observation  laboratory1  are  to 

1  .  Setup  the  experiment 

2.  Record  the  session  to  vi do* 

3.  Score  the  behaviors  observed  during  the  sessbon 

4.  Analyze  the  data. 

Practice  nuns  before  actually  attempting  lo collect  data  should  be  considered 
absolutely  necessary. 
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Preparation  before  an  observation  session 

For  troubleshooting  information,  pictures,  diagrams,  and  further  details  about 
each  piece  of  equipment,  see  Appendix  C:  Observarfon  Lab  Technical 
Reference 


1  General 

1 .1  .Reserve  the  AYVE  Observation  Lab  well  in  advance 

i  2,  Confirm  the  date  and  lime  with  ail  participants  and  others  involved 

1.3.  Open  HSIL  main  room  (GL-221 .  phone  ‘658-3138),  participant  (GL-221  a), 
and  observer  (6L-21 5,  phone  656-3134)  rooms 

1.3.1.  Check  lighting  a  nd  inspect  a  rea  To  r  cle  an  li  ness  a  nd  s  afety  issu  es 

1  3.2  Secure  any  equipment  papers,  etc.  that  couki  Interfere  with  your 
session  or  confuse  participants 

1 . 3.3.  Put  up  "RECORDING  IN  PROGRESS'  signs  and  inform  others  in 

the  area  of  your  work.. 

1  A.  Revievr  the  script  or  run  profile(s)  for  the  observations)  to  be  made 
1  5.  Brief  participants)  or  cameras,  microphones  and  general  lab  procedures 

1  6.  Ensure  consent  forms  are  property  Tilled  oul  and  on  file. 

2,  Set  up  panicipant  room  (GL-221^) 

2.1  .Arrange  furniture  in  accordance  with  script.  Pay  attention  to  camera 
locations  a  nd  try  to  keep  the  pa rtici  pants’  normal  movements  from 
blocking  the  cameras.  Tape  marks  on  the  floor  to  indicate  where 
part*ti  pants  should  stand. 

2.2,  Have  an  assistant  stand  on  \he  marked  spoi  while  you  preset  ihe  camera 
positions. 

2.3.  Provide  paper,  pencils,  and  other  props  (as  required) 

2  Alt  used,  turn  on  computer  (power  strip,  CPU.  display) 

2  5.  Participant  workstation  operations  check  (as  applicable) 

2.5.1.  Log  on  as  local  user. 

2.5.2.  Check  hardware  and  software  configure  tion(s)  and  network 

wnneeMy, 

2. 5 .3.  Log  o  or  otherwise  prepa  re  wo  rtc&tati  on  fo  r  participant  use . 

3.  Set  up  observer  room  {GU21 6) 
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3.1 .  Have  script,  notepad,  and  pens  available  for  taking  notes  during  the 
session. 

3.2.  Turn  on  power:  three  surge  protectors,  power  button  on  computer. 

3.3.  Visually  check  power  lights:  surge  protectors,  computer,  computer 
speakers,  printer,  event  logging  keyboard,  two  amps,  VCR,  two  monitors, 
camera  controller,  and  video  mixer. 

3.4.  Log  on  to  the  Observer  workstation  as  Noldus  Observer  (no  password 
required). 


O 
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Recording  a  Session  (Laboratory) 


The  instructions  starting  on  the  next  page  enable  you  to  record  the  screen  of  the 
computer  workstation  in  the  Participant  room,  the  live  video  feeds  from  the 
cameras  in  that  room,  and  the  audio  from  the  fixed  ceiling  microphone  and  the 
wireless  microphone. 

If  you  are  not  using  the  participant  computer  workstation  (for 
example,  when  doing  a  study  on  handheld  devices),  then: 

*  Skip  step  1  below 

*  In  step  3,  set  the  Observer  Camera  Viewer  window  to  [l/1 
si m  instead  of  |72|  size  (see  Appendix  C:  Observation  Lab 
Technical  Reference  for  instructions) 

*  In  step  4.2.4,  select  only  a  region  around  the  Camera  Viewer 
window  using  Camtasia  Recorder. 

For  troubleshooting  tips  about  each  piece  of  equipment  and  software,  see 
Appendix  C:  Observation  Lab  Technical  Reference. 
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1 .  Set  up  view  of  participant  workstation  screen  (only  if  using  ins  participant 

computer  workstation). 


1  i  Launch  application  Retd  onlyVNCte  participant 

1 .2.  Verify  that  you  see  the  screen  of  She  participant  compute*  in  the  VNC 
window.  The  VNC  window  should  be  positioned  in  the  upper  right  hand 
corner  of  the  screen. 

Z ,  Set  up  video  cameras  in  Participant  room, 

2.1.  This  is  one  of  the  more  complicated  steps-.  See  Appendix  C:  Observation 
Lob  Technical  Refere/tceon  the  camera  controller,  and  the  vsdeo  meter 
for  troubleshooting  and  details  on  other  video  setups 

2  2  Use  the  camera  controller  to  line  up  the  two  cs  meras  at  the  participant 

stalion  as  desired  Vou  should  t>e  able  to  see  their  output  on  the 

PREVIEW  video  monitor. 

2.3.  Use  video  mixer  to  set  up  your  video  feeds. 

2.3. 1  Raise  the  "NBar  to  the  top  position  and  center  the  joystick 

2.3.2.  Tap  the  "CUT  SOURCE"  button  (A  -  front  camera .B-side 
camera)  to  select  your  camera 

2  4.  Verify  that  your  desired  video  effects  are  displayed  on  the  PROGRAM 

video  monitor,  and  in  the  Camera  Viewer  window  in  the  next  step. 

3-.  Display  video  feed  from  participant  room  on  Observer  computer  screen. 

3  1.  Launch  T he  Obumr  CameriVlewer  (Neale! 

3.2.  Chech  Commands- Live  to  start  the  video  feed 

3  3  Position  the  CameraVI ewer  window  in  the  corner  of  me  VNC  window 
fpBCtiire-in-piciure’’  style),  or  to  Che  left  of  the  VNC  window  (if  it  is 
important  not  to  block  any  of  the  user's  screen). 

4,  Set  up  video  i  screen  recording  software 

4.1 .  Launch  Camtaila  Recorder 

4.2  Initial  Setup  of  CamtasJa  Recorder  -  you  should  always  record  a  brief  £5- 
1 0  seco  nd)  test  video  to  verify  your  a  udto  (sound  check)  and  video 
configuration,  (skip  to  step  4,3  for  subsequent  observations) 

4.2. 1  Position  the  Camlasia  Recorder  window  below  and  to  the  le  ft  of  the 
other  windows  (completely  out  of  the  way} 

4  2.2.  {O  ptional}  Venfy  confi  guration  in  Tools  >  Options  (see  the  Appendix 
C:  Observe  Uon  Lab  Technical  Reference  section) 
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4.2.3.  Capture  >  Wizard,  choose  Region  of  the  Screen,  check  Record  Audio, 
click  Next. 

4.2.4.  Click  the  Select  Region  button,  and  click  and  drag  from  the  upper 
right  corner  of  the  screen  down  and  left,  so  that  both  the  VNC  window 
and  the  CameraViewer  window  are  in  the  capture  region.  Click  Next. 

4.2.5.  Audio  device  should  be  set  to  SoundMAX  Digital  Audio,  (or  other  built- 
in  sound  card),  and  Audio  Input  Source  to  Line  In.  Input  Level  should  be 
at  maximum.  Click  Next. 

4.2.6.  Check  the  Disable  display  acceleration  box,  check  After  save,  play  the 
video,  and  uncheck  After  save,  edit  the  video.  Click  Finish. 

4  3.  Record  the  session  using  Camtasia  Recorder 

4.3.1 .  Click  the  Record  icon  ^  (or  press  F9)  to  start  recording. 

4.3.2.  Click  the  Pause  icon  1 1  (or  press  F9)  to  pause  the 

recording. 

4.3.3.  Click  the  Stop  icon  ^  (or  press  F10)  to  complete  the  recording. 

Be  careful :  clicking  the  Delete  icon  ^  will  completely  delete  your 
recording  session! 

4.3.4.  Save  the  video  file  (with  a  detailed,  descriptive  name)  in  a  specific 
folder  for  your  project.  Note  that  to  play  the  video,  you  must  have  the 
DivX  codec  installed  (available  on  the  desktop  of  the  Observer 
computer,  or  at:  www.divx.com) 
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Recording  a  Session  (Field) 

See  Appendix  D:  Portable  Observation  Laboratory  Embarkation  List  for  a 
detailed  list  of  the  equipment  that  comprises  the  portable  configuration. 

Digital  camcorders  (handheld  or  tripod  mounted)  and  digital  voice  recorders  are 
used  to  record  video  and  interviews.  Firewire  and  USB  connections  are  used  to 
transfer  video  and  audio  files  to  the  computer. 

Noldus  Observer  may  be  fully  used  on  the  laptop  only  if  the  USB  HASP  key  is 
attached. 

In  all  other  respects,  the  process  of  recording  a  session  in  the  field  is  similar  to 
that  in  the  laboratory. 

Some  notes: 

Take  copious  notes  about  what  was  recorded  on  each  video  tape  and  audio 
recorder,  and  exactly  when.  Each  time  a  video  tape  is  started  or  stopped,  the 
time  and  a  brief  explanation  should  be  recorded  on  the  tape. 

It  can  be  painstakingly  difficult  to  synchronize  multiple  simultaneous  camera 
angles  of  the  same  event  after  the  fact  (for  example,  to  set  up  a  picture-in-picture 
shot,  which  would  require  editing  with  Adobe  Premiere  Pro). 

It  is  much  easier  to  practice  a  session  once  (or  several  times)  before  recording, 
and  get  the  video  shoot  right  the  first  time,  than  to  do  a  lot  of  editing  the  video 
later. 
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Scoring  a  Session 


“Scoring”  or  “coding”  is  the  process  of  turning  an  observation  session  (usually  a 
video  file)  into  a  data  file  of  events,  event  times,  and  (optionally)  comments.  In 
traditional  usability  testing,  this  may  be  done  with  stopwatches  and  logbooks, 
which  is  tedious  and  prone  to  error. 


The  Observer  5.0  software,  by  Noldus  Information  Technology  of  the 
Netherlands,  will  be  used  to  assist  in  coding  the  observations  and  analyzing  the 
results.  Observer  is  capable  of  a  great  deal  of  customization  to  deal  with  different 
types  of  research  (such  as  time-lapse  recording  of  animals),  different  hardware 
(such  as  “old-fashioned”  video  tape  time-code  generators),  and  individual 
preferences.  In  this  guide,  we  will  present  many  settings  (without  further 
explanation)  that  are  either  specific  to  the  NPS  AWE  Center,  or  are  the  simplest 
for  the  research  conducted  here.  Advanced  users  should  read  Appendix  A: 
Advanced  Capabilities  to  learn  about  more  sophisticated  configurations,  and 
about  the  simplifications  used  for  this  basic  procedure. 


Full  details  are  available  in  the  Noldus  The  Observer  manual.  The  built-in  help 
files  in  The  Observer  are  easily  readable,  contextually  appropriate,  and  highly 
recommended.  Noldus  offers  telephone  and  e-mail  technical  support  that  has 
also  proved  to  be  very  helpful. 


The  Observer  can  be  installed  on  multiple  computers,  but  requires  the  presence 
of  the  USB  HASP  key  to  be  fully  operational.  Without  it,  it  will  operate  in  a 
restricted  mode  that  does  not  allow  changes  to  configurations,  or  the  use  of  video 
files. 


Definitions 

The  step-by-step  tutorial  in  this  section  has  examples  of  each  of  these 
definitions. 


Independent  Variables  Independent  Variables  (IVs)  represent  specific 

information  about  the  participants  and  the  scenario 
in  any  particular  observation  session.  If  you  don't 
use  tVs,  and  store  all  your  participant  /  scenario 
information  elsewhere,  you  lose  a  lot  of  analysis 
capability  built-in  to  Observer 
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Behaviors 

Events  that  are  recorded  during  a  scenario  are 
called  behaviors.  There  are  event  behaviors, 
which  occur  at  a  single  instant  in  time,  and  state 
behaviors,  which  have  a  duration  (see  Appendix 

A:  Advanced  Capabilities,  for  more  details). 
Behaviors  are  grouped  into  Behavioral  Ciasses. 

Subjects 

The  roles  (“billets”  or  "actors")  played  by  the 
people  simultaneously  participating  in  an 
observation  session,  such  as  “Team  Leader”  and 
“Rifleman”.  Very  often  there  will  be  only  a  single 
subject  (the  person  using  the  equipment  being 
tested),  and  so  subjects  will  not  be  set  up  in 
Observer. 

Scoring  or  Coding 

The  process  of  turning  observed  behaviors  into  a 
data  file  for  later  analysis. 

Projects 

A  project  file  contains  the  configuration,  all 
observation  and  data  files,  and  analysis  for  a  given 
experiment.  You  can  only  do  analysis  on  data 
within  the  same  project. 

Workspace 

An  Observer  workspace  is  a  way  to  organize  what 
projects  are  visible.  Use  different  workspaces  for 
different  researchers,  or  completely  different 
research.  The  Observer  system  in  the  AWE  center 
has  a  Sample  Workspace,  a  workspace  for  Class 
Projects,  and  separate  workspaces  for  theses  and 
other  research. 

Configuration 

A  configuration  should  not  be  changed  after 
starting  to  collect  data.  A  configuration  is  locked 
after  data  files  are  made.  If  you  unlock  the 
configuration,  you  must  remove  the  data  files, 
make  changes  to  the  configuration,  and  then  add 
the  data  files  back. 
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Noldus  Observer  step-by-step  tutorial 

1.  Configure  The  Observer 

See  chaplers  4  and  5  of  The  Observer  Reference  Manual  We  will  build  a  configuration  that  ends  up  looking  something 
like  this: 


You  will  only  need  to  do  this  long  configuration  process  once  for  your  project.  You  shodld  plan  on  not  making  changes 
to  Ihe  configuration  afler  you  have  started  collecting  data.  Its  possible,  but  likely  to  cause  problems. 

1 .1 .  Close  other  applications  and  launch  the  application:  The  observer  s.o 

1 .2.  File >  New  Workspace  (if  a  new  workspace  will  be  used)  or  Flle>Open  Workspace. 

Fi  le>  Works  pa  ce>  Properties  will  give  you  some  information,  and  allow  you  to  Change  the 
name  of  the  workspace. 

1 .3.  File>New  Project  Enter  a  descriptive  file  name  and  a  detailed  description.  Click  [Save]. 
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1 .4 .  Configuration >Setti n gs .  Set  as  shown  below: 


1  5.  Co nfigu ratio n>!ndependent  Variables.  Adjust  the  window  and  the  column 
sizes  so  that  you  can  see  all  the  columns.  Enter  information  relevant  to 
the  specific  observation  session.  This  information  will  be  used  later  in 
data  analysis.  Set  the  Type  to  Numeric  or  Nominal,  as  appropriate.  The 

"more  info"  button  with  three  little  dots  _J  in  the  Values  column  can  be 
used  to  restrict  the  entry  to  specific  values,  to  prevent  errors  (such  as 
spelling)  later  on.  At  the  very  least,  you  will  usually  have  an  Independent 
Variable  (IV)  to  identify  the  specific  participant  being  observed,  and  an  IV 
to  identify  the  researching  scoring  the  session.  Having  too  many  IVs  can 
be  slightly  tedious  to  enter,  but  will  cause  no  other  problems.  During  later 
analysis,  you  may  regret  having  recorded  too  few  IVs. 


Quemo  Project  -  Independent  Variable* 

ESQ 

Name 

i 

Values 

1 

PirticipartllD 

Numarit 

F norm  0  Id  99999 

2 

Scared  tnjf 

Mammal 

fakjgj,  Miller. 

3 

Aiga 

Nu  merit 

FramO  la  100 

<3 

Rink 

Mdminai 

NOiVMCO  £E1-E3J,  NCO  (E4*E6).  SMCt  .. 

5 

MGS 

Maminal 

6 

DijMlrghl  Condilmn 

Nominal 

Day.  Night. 

7 

Davies  Teal  ad 

Mammal 

3  * 

1 

1  6.  Conf igy ration  >Behavioral  Classes.  (Adjust  the  window  as  necessary) 

1 .6.1 .  The  Observer  calls  any  event  that  you  observe  a  Behavior.  Behaviors 
are  grouped  logically  into  similar  Behavioral  Classes.  Enter  your 
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Behavioral  Classes  3S  Shown  below. 


1 

' 

Name 

II 

Pllllliull 

1 

Everts 

a  cubcJmofamran  trerti 

J 

i 

Enas 

poss  tile  error  everts 

Nomrrt 

jhprt  WMgjCttifi.  wfca  crt*h 

J 

3 

everts 

NW  nd 

Mmmert,  LJWjxpetf  «l 

□ 

4  * 

1 

1 _ 1 

1 .6.2.  Enter  Behaviors  as  elements  within  each  Behavioral  class.  Clicking  on 

the  more  info  button  in  the  Elements  column  allows  you  to  enter  the 
Behaviors  in  each  class. 


Leave  Modifier  Classes  set  to  None. 

Always  allow  The  Observer  to  set  the  behavior  code  for  you 
Change  Properties  to  Event  for  instantaneous  events,  such  as  error 
behaviors.  This  step  is  easy  to  miss! 


*  Ten  project  ■  Errars  ■  EMiaviori 


lUme  |  |C(K»e 

Moddlcl  CUM  1 

|  Maddiei  a***  2  |  Pistil**  | 

1 

2 

m  t  HMF  dm  neorrertty  (wOrtfl  rtt.x 

Wj8rrM  endkW'*i**»rr'Xl 

h 

More 

None  f Lrtrt  \ 

3 

VV1  LI'J  dL'JWI  i.J'j  Lf  rUMK'Ol  1^ Jion  Wt  irmkif  4  K  Lwrrji.1 

VfI 

None 

None  I  &rtrt  J 

4 

syston  crosTi  Psv  w  r«fc-d  not  due  lo  user  vtv 

None 

None  \  Evert  / 

5  * 

! 

1 .6.3.  You  should  use  State  Behaviors  for  scoring  events  that  have  a  specified 
duration,  such  as  time  to  complete  a  task.  The  participant  must  always  be 
in  one  of  the  states  in  every  state  behavior,  such  as:  ON  Task  1 ,  OFF  Task  1 . 
This  is  easier  to  work  with  than  separate  events  like  START  Task  1 ,  END 
Task  1 ,  because  The  Observer  will  automatically  calculate  the  duration  you 
are  in  a  task,  error  condition,  etc. 


1 .7.  Configuration>Check  Configuration.  The  Observer  will  check  your  configuration  for 
completeness  and  conflicts. 

1 .8.  Configu ration > Review  Configuration.  File>  Print.  Print  a  copy  of  your  configuration  for  use 
during  scoring. 
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2.  Star!  an  observation!  session 

We  will  set  up  for  an  observatia  n  session  (from  a  medta  file  recorded  earlier) 
that  looks  something  like  this: 

IjfcITk*  G!m ■*>"#■  f  ■  bhurvilSArt  !\kpdiih  Dime  Prdjarl  _  [ 

-_!■  '.-p-i  ujrj  wJi  hVl^  f  ,1-i-s  -■  -rf 


2  f  .Launch  and  con  Figure  Ihe  Observation  Module:  Observe  tian>  Slart 

Observing.  Vo  u  will  only  need  to  do  this  bng  con  figuration  process  once 
for  your  project.  Scoring  the  videos  will  proceed  much  more  quickly  after 
the  first  lime. 

2.1.1.  IF  you  Ihink  Itie  Observation  Module  settings  are  incorrect  To  r  some 
reason,  you  can  use  Customae^Reset  Custom  izalion  {lo  sot 
everything  to  the  defaults)  before:  setting  the  customization^  below. 
The  Help  button  in  each  of  the  following  wi  ndows  ha  s  excellent 
descriptions  of  Ihese  settings 

212  View^Codes,  Monitor  Timers,  and  Video  Control  should  be 

checked,  others  a  re  optional  (Monitor  en<f  Video  Control  may  not  be 
avails  bra-  until  Timing  is  satin  step  2  1.4).  View^System  Status  can 
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be  useful  to  dreptey  Independe  nt  Variables  or  other  inform  a  Cron,  as 
shown  in  the  screen  sl»t  above. 


2  1 .3.  Arrange  and  si^e  the  windows  conveniently,  as  shown  above 


2. 1 .4.  Cuuoml  ze  >TI  min  g.  Set  a;s  shown  below.  You  shoutd  always  use  (he 


&>m  Timing  settings  tor  a  !■  ofrscrvfriipnsi 


Twin* 

?o 

EvM  fnr¥Kj  liri-Pd  <hi 

r  lm  toUTJifcrCfe-Lhl 

r-  iM^B^Wh^l^lKiHlip/^Pq 

r  Oadu-on  Tapf 

i  Ewfln'*r*jff«*r«*L4»frFit 

r  Tire* 

T.mo  «|  E-j*  jl  :-f,isiLr 

r 

■  A^ndTBfifnwVdwTir.* 

r  ijfmm: 

Q*  j  UffPri  | 

if  sol  So  ‘Live. "  you  w'J  i»  twabfo  to 

vhS?o  sefflvgs  ir»  /aftsr  steps.  of  fiaatf  r»«to 

fifes 


w  ser  ao  ¥Atoays  Zero M  af  me  Sferr  of 
O&serve r*pn  you  «n  nave  problems  wirn 
Srn^  codes  ff  you  5&1T  m  the  middle  of  a 
video  and  men  rewnd  fl  past  ‘feera H 


2.1.5.  CustomHe>Qperatlon  Set  as  shown  below 

Ofr«r*Tfrwi 

ElrtioifMna^fTl  Von  nLHni  jni  '■biirvot-crt 
T  Ufcv 


T  QpliWi 

■  fif-fYn 


r  !tFiWT*f*IWl*tfl  Vftrt  iTN*i  ■flfti'JlM  fr-rfHE* 

■  Biai 

r  cbm 

r  Afr-iryi 


V  Fwfrfcj  Contimebamj  Dwl  Obm-vaiof ■ 


I 
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2  1  6.  Cmtomb» wi  ndow  pwpnri«t>C&dw ,  Sel  as  etiown  below 


2  1  •  7._ Cui tomlM  > Wl  ndcw  Properties*  Event  tog.  Set  3S  SllOWn  below: 

Ey*nrt  Lo-|  Ptop*rMii  ■■■  jm 


Opfer-ni  Ctturr:. 

P  SaraT#™ 

p"  £ 


Eitjto  Tirr*1  Ai 

>i  fifciw"  d  T  r«  3Wi  "vn  i  ■  d:i; 
yuw  TWw  H  rm  ti 


Timer  opfrfirt 
0ftfy  *ppte£  to  Ua-ATU 
am  rag  video  fcrpc: 


21  ,&Ci»1joinfze>  Window  Properties  >Tlnnra  Setgs  sh-own  feelrtW 


Timin  Prepwtki 


m 


T  nns  D  i  uli-nVii 

^  tjTf*  I  ITT  Hi 

r? 

>  b^.>:  »J  '  i'll  ■ 


I  I'shrcTiih-  i  tfiMI 
P  £^:r**,iP-y  T  Pte 

fprt*fc  r+*.|  Ljj !-[.!■.■ 


T  tr-H  r-nifr  Lfc-f 

3 


vjTM^I 


2.1  .S.  Customize > Window  Properties > Video  Central  S-e-E  to  EXPANDED. 
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2.2.  Customize:* Data  Storage.  Set  as  shown  below: 


2.3 

2.4 


.  Flle>Open>Medla  File.  If  this  is  unavailable,  check  the  Timi  ng  selling  in  step  214 

.  Data>5tart  Observation.  You  will  star!  an  observation  to  score  each  separate  observation  (that  is,  a 
recorded  video).  The  Observer  will  automatically  supply  a  name  (because  of  the  Auto-Name  setting  in 
Customize  Data  Storage]  A  Title  allows  you  to  enter  a  more  detailed  description 

Start  Observation  m 


|Demo00D1 


£ils  Name: 

T11®'  jkfy  first  oboervafion  ■505' 

Volume  Name. 

Llocfio,FI*  Nam»i  |  Darn  o  Movie  F.le.myy 

Observation  Mad  a 
£wre  EwslingVidso 

r  -Sprite  while  ;  p—rlnj 

r  Beyiew  Video 


~zi 


Qruwse  | 


Exiemol  Application 
\~  UDD^ammardLinc 


|  QK  |  Cancel  |  Hfrlp  | 


2.5.  independent  Variables.  Enter  the  values  of  the  IVs  for  this  session. 

2.6.  Finally,  clicking  this  start  button  will  start  the  observation  session: 


2.6.1.  As  the  video  plays,  score  behaviors  by  clicking  the  appropriate  box 
in  the  Codes  window.  Use  the  Video  Control  window  to  pause  or  rewind 
the  video. 


© 
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2  B.2.  To  enter  a  comment,  double-ctick  in  the  cemmems.  column  of  the 
Event  Log  You  must  press  [Enterl  when  done  typing  a  comment 

2.6.3.  To  edit  a  scored  event: 

2£ .3.1  Pause  the  video  using  the  video  Control!  window 
2  6.3.2  oiti>Edit  olwe-rraiion,  and  edit  the  event  log  as  necessary . 
2.6.3. 3  Data*  uncheck  Edit  Observation  to  return  to  scoring  mode. 

2.6  4.  When  scaring  the  video  is  complete,  choose:  Data>End  Otoe  muon 
2.7  To  go  back  and  edit  an  existing  observation  file  alter  ending  an 

observation,  choose  Dmra>5ttri  observation  (as  in  step  2.  A)  and  enter  the 

name  of  the  exlstl  og  observation  file  -(instead  of  a  new  file  name).  If 

prompted  Tor  "Media  File  Position",  do  not  check  "Use  Time  in  Medra  Fiel 

as  -Start  Time  for  Data  Fife" 


Continue  stoning  the  observation  normally  Vo  u  can  add  new  events  to 
the  previous  file,  or  go  into  Edit  Observation  mode  to  edit  existing  events. 

2.8.  After  you  have  completed  scoring  all  the  videos,  dose  the  Observition 
Module  End  return  to  the  Mam  Module:  Fife>«etum  to  Main,  Vtfien 
prompted  to  $*ve  the  customization  rile,  crick  [OK],  The  customization 
saves  your  Observation  Module  settings,  window  arrangements,  etc. 


Q 

m 
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Analyzing.  Data 

You  can  analyse  data  collected  using  No  Id  us  The  Observer  in  two  general  ways 
exporting  raw  observations  into  a  dedicated  statistics  package  sucfo  as  SPSS,  S- 
Rus,  or  Excel,  or  you  can  use  Th a  Observer's  built-in  basic  analysis  tools  You 
tan  also  then  export  that  analysis  to  another  program.  For  most  data  analysis, 
you  wilt  need  to  consull  she  documentation  a  bo  ut  the  specific  ways  you  want  to 
organs  and  analyze  your  data. 

Exporting  data  from  The  Observer 

Tfie  Observer  row  data  files  are  normally  found  in  the  directory. 

C:\Oocurmnts  and  Setting s\AII  Usere  \  No  IdtisM  he  Obserwt  Works  twees  \  Projects'*,  Your 

/^pKfMime\DaU  Files 

You  should  not  move  date  Hies  {or  any  other  hies)  from  Iheir  default  Ideations. 
Data  hies  have  the  ,odf  fire  extension  (exampte  Dtmeoooi  but  are  normal 
text  Hies  Copy  the  data  files  you  want  to  another  location  (like  a  folder  on  your 
desktop),  end  rename  them  with  a  ,txt  txfenslo  n  They  no*  can  pe  opened  with 
Notepad,  Word,  etc  Do  nol  edit  or  alter  Ihe  original  _odf  files. 

It  is  probably  easier  to  use  Observe/  for  organizing  the  raw  data  and  pe /forming 
basic  stashes  {see  the  roe* f  section)  before  exposing  it  10  another  program. 

Using  The  Observer's  analysis  tools 

Data  analysts  in  Tfre  Observer  is  a  multi-step  process  Firs!,  you  select  the  data 
to  a  nalyze,  and  then  you  can  view  the  data  graphically  [Time-  Event  View ),  o  r 
perform  elementary  statistics  or  reliability  analysis  on  it. 


1 .  Elem  en  la  ry  Statist  ics. 

To  understand  alt  the  op!aons  an  Ihis  function,  try  generating  a  number  of 
different  slalistics  sheels  lo  see  ihe  differences 

1  1  .From  The  Observer  Mam  Module  {rot  the  Observation  module  -  see  slop 
2 .2  above)  Analysis  >  Elementary  Statistics  >  Open  Analysis  Profile... 

This  win  edil  the  currenl  Statistics  Profile,  which  defines  what  statistics  wil  I 
be  computed,  a  nd  how  111  e  output  will  look.  You  can  create  New  Analysis 
Profiles  and  save  different  settings  in  each  <u«  active  Dai*  Piofilo  is  fine 
for  Ebe  firs!  use.  Ft  you  don't  already  have  one,  The  Observer  will  create  a 
generic  data  profit  fer  you  that  includes  a^  the  data  coited-ed  {usuaity 
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what  you  want). 


1  2.  On  the  Statistics  tab,  choose  the  statistics  you  are  interested  in.  Durations 
witl  only  apply  if  you  have  used  state  behaviors  or  have  defined  time 
intervals  between  certain  events, 

1 .3.  On  the  Layout  tab,  turn  off  the  Modifiers  and  the  Subject,  since  you  will 
normally  not  use  either. 

Use  the  up/down  and  left/right  arrows  to  change  the  arrangement  of  rows 
and  columns  on  the  output  statistics.  You  should  probably  run  some 
different  layouts  to  figure  out  how  you  want  the  Statistics  Results  to  look. 
Turn  on  Interval  only  if  you  are  using  state  behaviors,  or  have  defined  time 
intervals  between  certain  events. 

Turn  on  any  independent  Variable  values  you  would  like  to  show  on  the 
Statistic  Result  output. 

Change  sheet  Per  to  <one  sheet>  to  put  all  the  data  on  a  single  sheet, 
instead  of  on  separate  tabs.  Otherwise,  choose  how  you  want  the  tabs 
broken  out. 
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1 .4.  Click  Run  to  create  the  Statistics  Result  from  this  Analysis  Profile.  Note  that 
you  can  select  cells  (using  shift+ click  to  select  multiple  rows)  in  the 
Statistic  Results  output  and  paste  them  directly  into  Excel  and  other 
applications. 


O 
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2.  Time-Event  View.  A  Time-Event  Profile  is  a  quick  way  to  look  at  data 
graphically,  or  to  organize  it  to  export  to  other  software. 

2.1 .  Use  the  Layout  Tab  to  choose  a  Time- Event  Table  or  Time-Event  Plot.  In  the 
examples  below,  sheet  per  is  set  to  cone  $heet>  so  that  all  data  is  shown 
on  a  single  plot  or  table  Click  kirn  to  create  the  table  or  plot. 


Tim*-Event  View  Setlingf 


m 


Date  Input  Leymrt  Objq  Output 

Mode 

r  Time€vernleb!e 

(■  [Tirne-E^TifJElc^ 

Sari  order 

Columns 


Sheei  per 


PriEiily 

:| 


F  Ob^ruarian 
I”  Subject 


koine  sheeo 


Merge  Pam  fer  one  interval 

r  I  '!f"n  p  h  (■ ;  t,  .iiir/rtiri  rM«*r- 


P^i;n  | 

I  Bun  J 

posa  | 
Cancel  | 

mp  I 


2.2.  Sample  Time-Event  Plot.  You  may  have  to  resize  the  window  to  see  all  the 
data 


^  Dame  Project  ♦  Tkna-Evant  Vlaw  -  Time-Evant  Pssuli  (3) 


III 


FMi  JtJtor  Jl  CIbm 

Ofcitrvtfut 

E^nts 

DempWOI 

UE-mouwi^ 

Evans 

DfrmOKMJ 

Eirori 

OemoDOfil 

-  i- 

DtmOM? 

Efrflrt 

OumoDOOS 

Wsc 

OamaDQfll 

Miec 

D*rr»DOJ2 

Mrtn 

nwwnnraia 

^BQ 


i 

i  i 


Tune  i» 


*  |>  ft A II  Data  j( D ata  Pi u ItefcfltKitfs a  Profile 


Letierd 

8*twor 

CokHifWe^ri  Bslvtuior  Colgu'FyiflflJn 

J  sm  ::.iii  |  ni  r  i  n  J 

wilnCl 

event? 

J 

svjWMrteraah 

U  1 

un^TW^Pd 

j 

j_tri 
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2.3.  Sample  Time- Event  Table.  This  data  can  be  cut  and  pasted  into  other 
Windows  applications.  or  exported  directly  to  .XLS  or  CSV  files. 


Kl Demo  Project  *  Time-Event  View  -  Time-Eve. , .  L  I’DQ 

Start  Time 

Behavioral  Class 

1  Behavior 

]  Observation 

1- 

116 

j  Bents 

pvfinrl 

PemoOrai 

1,80  Brents 

evqnil 

PemoOQQl 

7.37 

Bants 

Rvanl  1 

CpmoDrOI 

5  72 

Events 

pvpnll 

fifin-nDOte 

— 

Bents 

p'.-enil 

Pemq0002 

6.00 

Bants 

evenlt 

DBmoDOGE 

6.08 

L  vents 

erenH 

DemoOOEE' 

6.34 

Birins 

p^rill 

Demo0C05 

6,36 

Events 

evenO 

DemoDOTJJ 

676 

events 

avanFl 

|DamOD003 

8.20 

Errors 

wrong  action 

CeroaDDCG 

'  jlI 

«  |  ^Atl Data ^ Data  Profile^  * 

1 

> 
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L  Data  Selection. 

Use  Dale  Profiles  to  restrict  or  filter  your  analysis  to  specific  subsets  o  fthe 
data. 

3  t .  From  Th&  Ohs&rver  Main  Module  (not  the  Observation  module  -  see  step 
2.2  above }:  Data  >  Data  Profile  >  hiew._.  Give  the  date  profile  a  name 
Note  that  a  d£ata  profile  window  has  items  in  rows,  and  six  tabs  along  the 
bottom; 

-  g  D»niQ  PrDj*ct  -  DaU  Protik  Cl)  -  Dali  ’Stticllori 

L  ;=*A«4-tO 


J^f\lnd#p^ndsnl  Vanablns  l  ■>!■,■.  &iLf«c:L  frt,\ i-.  {  Mud  r  /  e:vj  ■,  / 1 

3.2  Edrt  a  row  on  the  appropriate  lab  to  create  (he  filter  Tor  your  data 

3.3.  Once  you've  created  a  filter  for  your  data  using  a  Data  Profile,  you  can 
select  that  Data  Profit*  in  Efemenrarv  Stailsiks,  Time- Event  Plot,  and  other 

analysis  funclions. 
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Video  Equipment:  Other  Capabilities 

The  video  equipment  in  the  observation  laboratory  can  be  used  for  a  variety  of 
other  functions,  including: 

•  Capturing  VHS  video  tapes  to  computer  files  or  to  DVD. 

•  Recording  (“burning”)  computer  media  files  onto  DVDs  that  can  be 
watched  on  standard  home  or  computer  DVD  players. 

•  Recording  training  sessions  from  multiple  video  angles  and  direct  screen 
capture  to  computer  media  files. 

Capturing  a  VHS  video  tape  to  a  computer  file 

1.  Complete  the  standard  system  start-up 

2.  Put  the  videotape  to  be  recorded  from  into  the  VCR.  Rewind  the  tape  to  the 
starting  position. 

3.  The  output  cables  from  the  VCR  go  to  ‘video  and  audio  input  #4’  on  the 
Video  Mixer.  The  audio  output  of  the  video  mixer  gets  plugged  directly  into 
the  Canopus  board  on  the  CPU.  You  will  have  to  disconnect  the  current 
audio  inputs  in  order  to  do  this.  Note  what  they  were  so  you  can  put  them 
back  again  when  you  are  finished. 

4.  Set  Video  Mixer  to  input  source  D,  which  corresponds  to  input  #4.  If  the  video 
input  is  not  displayed,  see  the  Focus  MX-4  mixer  manual  to  check  the  video 
“Route”  setting  for  input  4. 

5.  On  the  computer,  launch  the  Canopus  Media  Cruise  application. 

6.  Set  the  Canopus  Media  Cruise  to  video  recorder  mode. 

7.  Click  “Format”,  and  set  it  to  “MPEG 2”  and  “Program  Stream”. 

8.  Click  “Parameter”;  enable  and  enter  the  max  video  capture  time  (i.e.  the 
playing  time  of  the  video  tape.)  You  can  stop  the  recording  earlier,  but  it  will 
run  all  night  if  you  don’t  put  in  a  time. 

9.  Click  the  folder  icon  on  Canopus;  this  is  also  known  as  the  “Capture  File 
Configuration".  Enter  a  file  name  and  location. 

10.  Press  Play  on  the  VCR  to  preview  the  audio  and  video  levels.  When  these 
are  satisfactory,  rewind  the  video  to  its  starting  position. 

11.  On  the  Canopus  control  panel  on  your  computer  screen,  click  the  button  with 
the  red  dot  to  begin  recording,  then  immediately  press  play  on  the  VCR 
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12.  Click  the  button  with  the  white  square  to  stop  the  recording. 


Burning  video  files  to  DVD 

1.  Launch  Sonic  My-DVD  Studio  Deluxe  Suite. 

2.  Choose  Video,  then  Create  Project,  then  DVD. 

3.  Click  the  “Add  files”  button,  then  select  the  file  that  you  want  to  burn  onto  a 
DVD. 

4.  Change  the  background  and  text  as  desired.  This  is  the  time  to  add  subtitles, 
trim  or  crop,  etc. 

5.  The  quality  setting  should  be  set  to  “Fit  to  DVD”. 

6.  Insert  a  recordable  DVD  into  the  computer.  (Note:  the  CPU  in  the 
Observation  room  only  works  with  ‘DVD  plus’  discs,  such  as  ‘DVD  +R’  or 
‘DVD  +RW.  It  will  not  work  with  ‘DVD  dash’  discs,  such  as  ‘DVD-R’  or  ‘DVD- 
RW’. 

7.  To  create  the  DVD,  click  on  the  big  red  “Burn”  button. 
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Appendix  A.  Observer:  Advanced  Capabilities 

The  following  are  the  main  restrictions  that  have  been  used  in  this  Noldus  The 
Observer  5.0  configuration  to  simplify  the  tutorial  in  this  guide: 

•  Only  event  behaviors  -  no  state  behaviors 

*  One  subject  only  (and  so,  no  channels) 

•  No  modifiers 

*  No  live  scoring  -  scoring  from  video  only 

•  No  use  of  the  event-logging  keyboard 

*  Recording  video  using  only  Camtasia  and  Camera  Viewer,  rather  than  the 
Canopus  MPEG-2  encoder  card. 

The  Observer  is  capable  of  much  more  sophisticated  configurations.  More  in- 
depth  research  will  benefit  from  using  features  such  as: 

State  behaviors  are  useful  for  coding  tasks  in  usability  analysis.  State  behaviors 
have  a  duration,  while  event  behaviors  are  single  points  in  time.  State  behaviors 
in  a  behavior  class  (e.g.  "posture")  are  mutually  exclusive  and  must  define  all 
possibilities  (such  as:  standing,  sitting,  prone,  other).  Every  subject  must  be  in 
some  specific  state  behavior  in  each  state  behavior  class  at  all  times.  Some 
usability  researchers  will  use  states  to  identify  tasks  being  performed.  Analysis 
on  the  duration  of  events  is  easier  when  using  state  behaviors  than  with  event 
behaviors. 

You  can  coerce  two  event  behaviors  to  act  like  the  beginning  and  end  of  a  state 
behavior,  to  get  data  on  duration,  but  it's  easier  to  just  use  state  behaviors. 

Multiple  subjects.  Subjects  are  the  roles  played  by  the  participants,  such  as 
"Primary  User,"  and  "Assistant".  Subjects  would  only  be  used  if  there  was  more 
than  one  person  simultaneously  participating  in  a  scenario. 

Modifiers  can  be  used  to  refine  behaviors,  such  as  a  "walking"  behavior,  with 
modifiers  like  "slowly"  and  "quickly",  rather  than  many  separate  individual 
behaviors,  such  as  "walking  quickly"  and  "walking  slowly".  Using  modifiers  allows 
for  more  options  during  analysis  to  group  and  categorize  the  data. 

You  can  code  behavior  live  while  it  is  being  recorded,  and  then  go  back  to  the 
video  file  and  refine  the  coding  later.  This  might  save  time  for  experienced  coders 
doing  a  lot  of  observations,  but  is  generally  too  much  to  do  at  once  for  beginners. 

The  event-logging  keyboard  can  be  customized  to  the  specific  events  used,  and 
labels  attached  to  the  keys.  The  jog-shuttle  dial,  used  to  control  video  clips,  is 
extremely  helpful.  There  is  a  "programming  switch"  on  the  upper  right  side  of  the 
X-Keys  keyboard  that  should  show  red  to  program,  black  to  use.  You  may  have 
to  exit  other  applications  when  programming  the  keyboard,  and  sometimes  even 
restart  the  computer  to  get  the  macros  to  “stick”. 
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Event  Summaries  (Wofdus  The  Observer  Reference  Manual  section  S  4)  can  be 
helpful  far  gene  rating  quick  reports.. 

Reliability  analysis -consistency  betwe  en  different  carters,  or  a  single  coder 

scoring  in e  same  video 

Lag-sequenlial  analysis  —  identifying  events  "that  tend  to  follow  or  precede  other 
events,  and  even  "chains."  of  events. 

The  Canopus  MPEG-2  encoder  card  is  a  fester,  higher-quailty  way  to  record 
video  if  only  wtfeo  from  the  cameras  is  required,  The  Canopus  system  (card  and 
Mediaerur&e  application)  ^aon  at  rooon d  events  ort  the  aarem.  sutii  as  the-  VNC 
screen, 

1 .  Co  nflgu  rat  Ion  ■  Subject 

Subjects  will  cmy  be-  used  if  there  are  munipte  peojs*e  mreraehng  m  »  scenario  at  once 

will  gffen  net  be  used  if  mere  is  enfy  er>e  n-g  configurgGign  i@  ngge&siry  rgr 

sutjjecls  ir  subjects  are  used.  grve  them  lhe  narre  oT  the  role  poarton  or  i>!let  they  are 
performing  in  E  he  scenario  If  ne  cessafy,  ihe  individual  parlKi  pant's  actual  name  should  be 
stored  £6  ihdetendG-nt  variables,  hoi  as  lhe  subtect  name 


1  1  C onf I g ui atton  ^Cha mils-.  Channels  are  required  &f  there  ia  more  than  one 
sublet.  In  your  scenario,  subjects  may  be  allowed  to  perform  all  defined 
behaviors,  or,  by  limiting  the  available  channels,  subjects  may  be 
restricted  to  only  specified  behavior  classes  This  is  mainly  to  prevent 
data  input  errors,  and  group  subjects  and  behaviors  more  topically. 

Topics  to  ignore  in  using  Mcldus  The  Observer 

1  Coding  from  ofdeofape.  In  the  pa  si,  analog  videotape  recorders  with  'time- 
code  generators"  were  used  in  behavioral  research  Much  of  the  video  and 
timing  configuration  in  The  Ofrse rvor  5.0  deal  with  these  situations,  and 
should  be  ignored. 

2  interval  (states  or  events)  recording  Situations  calling  for  thesfi  types  of 
recording  methods  are  uncommon,  As  opposed  to  continuous  recording 
(which  we  use),  Ihese  generally  apply  lo  time-lapse  or  long-term 
observations. 

For  more  Information  on  these  capabilities,  see  the  Mold  us  The  Observer  5.0 

documentation.  Also,  Noldus  has  been  very  helpful  in  email  and  telephone  tech 

support. 
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Appendix  B.  Observer  Lab  Components 


Observer  station  components 


O&servw  workstation 
computer 

Cymensran  9400  F4  3.  ttht 

TOGS  HD.  1GB  PAM.  Ptwp  s  DVDOtiQi  CD-RWCVD+RW 
Rewire.  ATI  R&Mon  XJO 0  $wb$ 

wild  special  hard-vefe  and  software  described  below 

Even!  logging  keyboard 

Pf  fngirmnrv  x-^s  $£ 

Camera  joystick  control 

re«o™ri7«  cp-^tv-  vcj 

Uid&o  mufei 

RjcmErtuncaimrts  J4fc4  SN&wsii  n#ir&MJAMi(«u 

uuccftl  ^  fatwKi  to  con  rmhuah  and  fVmwflrp  [town  Ido  ft  nl 

Mis  tfww#  1odJvr:Js  &«n 

mmiDV  VCR  <po  liable) 

smpGV-Diooo 

Wireless  microphone 
receiver 

Ajfdan  20QR 

Mobile  device  camera 

Kern  Ggasystem  RXCHmci  cofrpany 

receiver 

Observer  microphone 

Stour*  MX472D/C 

Mic  amp 

PrcSonus  MP20 

Speaker  amp 

tj$  AU&Q  UtflrffWYntf  R-  J2 

Speakers 

Radio  Shack  400-4  TCI- 

EJflWvtf«  PhOfiilors 

Show  the  preview  and  program  euipute  of  (he  views  miner 

One-way  mirrored  glass 

49'  X  36' 

Observer  workstation  { special  hardware  and  software) 


Motdus  HASP  USB  serial  key 

Mum  be  preset  lo  use  ratios  Cfcseiw  sort were 

KtfiM  r^mtor  QBiKO-K*22- 

OdAA 

Noldus  The  Observer  vS.O 

SoH^a  re  a  pclcs  tion  for  coding  gnd  necortfing 
behavior 

Canopus  MPEG  Pro 
MVR10OTSX  PCI  card 

Video  encoder  doard  ■  reel-time  MPB3-2  encoder 
ifro m  aralcsg  nr  DV  input)  TaKesIhe  acVrjar  output  cf 

1  he  video  nnxEr  CtnuiuiiiinSinteH  i40fi)9i*<Mft 
tut  dp  «ii  nnd  imwoi  hi  tflE^-iicaa LsansaiiLui 

Canopus  Media  cruise 

Progtam  so  encod*  arkalog  video  input  to  MPEG-2 

Ntt 

Canopus  MVR  OV-MPEG  File 

Used  to- convert  rgw  Crv  tne»  to  MPEO-2  format 
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Converter 


Euresys  Picolo  PCI  card 

Video  capture  board  (PCI  slot  2).  Allows  screen 
capture  of  the  program  output  from  the  video  mixer 
(the  Canopus  board  does  not). 

Noldus  Camera  Viewer 

Viewer  for  preview  video  in  to  Picolo  Card  from  video 
mixer 

Sonic  MyDVD  Deluxe  Suite 
6.1 

Software  DVD  player/recorder.  Watch,  edit,  and  burn 
DVDs  (cannot  use  PowerDVD  or  WinDVD  -  they 
conflict  with  The  Observer) 

Techsmith  Camtasia  Studio 
2.0.3 

Record,  edit,  publish  videos  of  PC  desktop  activity 

RealVNC  4.0  (in  viewer  mode)  Observe  /  control  the  participant’s  computer  screen 

X-keys  Macro  Works 

Controls  the  event  logging  keyboard 

DivX  codec  5.05,  DivXplayer  Video  codec  and  player  for  highly  compressed  DivX 

2  6  video  files 

Adobe  Premier  Pro  1.5 

Professional  video  editing  software 

Participant  station  components 

Participant  workstation 
computer 

Windows  2000,  512MB  RAM,  2Ghz  Pentium  4.  Two  18  GB 
drives.  Could  be  any  computer,  but  must  run  RealVNC 

Server  (which  requires  administrator  access  to  install) 

Pan  /  tilt  /  zoom  cameras 
(2) 

Canon  VC-C4  and  Canon  VC-C50i 

Ceiling  microphone 

Share  MX200  omnidirectional 

Wireless  microphone 

Azden  31  LT  freq:  A4  171.905  Mhz 

Mobile  device  camera 

Noldus  MDC-03N 

Office  furniture 

Desk,  conference  table,  dry-erase  white  board,  chairs 

Individual  military 
equipment 

Helmet,  flak  jacket,  MOLLE  harness  and  cartridge  belt, 
rubber  M-16s 
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Appendix  C.  Observation  Lab  Technical  Reference 

The*  most  commonly  re  fen e  raced  Ups  and  troubleshooting  facts  are  highlighted. 


Observer  Station 

Observer  workstation 

computer 


Computer  name:  B2W6EM51  inol  on  NPGS  dornainj- 


IP  Address.  fS  inlie) 
SUM 


Gateway : 

DNS  servera: 

MAC  Address: 

Del  Asset  Tag: 
Adminisiririo*  password: 
normal  username: 


131,120.147.  u: 
2S5.2SS.2S2. 0 
131.120. 14 J. 1 

191 1.120*10*4  0 
131.120.16,41 
0D-ai-ll-3D-D2-22 

2MSZM51 

WOLdUJZHSl 
Woldua  Observer 


password :  {wjw} 

Back  ad-up  fGhosI  ung^a)  by  Jim  Siou.  SSe-2499,  on 
IE  Fab  05.  Image  files  si  wed  on: 


Camera  joys  tick  control 


H,!SM  =!  !>y  unpliMjcjaKp . I  | ill .;:j |  Li;i<:k  in  if  or . I  tin: 

cameras  is  turned  oil  or  dher  problems  occur. 


First,  press,  ihe  button  -corraspwding  to  the  camsra 

you  wanl  Id  control,  then  use  Ihe  joystick,  zoom,  or 

locus  oortlroit.  Tlwe  *  a  brtof  defty  before  the 
camera  will  begin  id  move, 


how  ihe  s*i  ballons  dawn  ten  4  »eond*  io  »i 
pnulft  (JiPWilfln,  fwiiSL  fl  Ad  2W*n},  SwnHi™.  you 
migfrl  want  to  lum  the  aula-focus  OFF,  so  thal  i 
doesn't  keep  changng. 


Video  mixer  Focus  MX 4 


Controlling  this  mixer  can  be 
OOtffuiiitjl.  YChj  vnll  lileoly  Ivrvc  Id 


There  are  atoays  two  video  sources -selected  {vMh  Ihe 
two  raws  of  buflons  at  the  to  p  of  Itis  mi*  erj  -  She 
"Gurrenl"  (or  "cuD  source,  and  Iha-  "nenl "  source 


Configuring  video  inputs 

If  you  change  the  inpul  cabling  M  all  ctib  cameras. 
MDC,  VCR ,  ale.) ,  you  must  set  them  up  by  pressing 
the  [ROUTE]  buElon  (cb  2  in  manual).  Note  whether 
you*  inputs-  uso  llifi  S*vxfoo  or  INj  RCA-Style 
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consult  the  ftJl  manual  ror  any  composite  pets 

problems  you  eapenenoe 


Vqy  d*  notify  diEplay  two  inpuiE 
sffnutlafieausiy  (even  though  there 
a  re  tou  r  dtffe*ent  input 


Useful  effects  include  ptctyre-in- 
pieture  {PiP),  t&p-icMbcitlOCTi,  and 
w^by-side  cute 


CtfntAguffcng  lAdeo  autpuls 

Press  the  [DISPLAY!  bM>n  muHspb  li^es  to  see 
diPTefeni  view®  or  the  camera  i*!ecncrp6  on  the 
PREVIEW  monitor,  wnch  rs  used  for  dbnfi  gurtton 
The  ad  uei  output  to  the  oompuief  ts  shown  m  the 
PROGRAM  'monitor 


Baste  operation  -  always  co  this  firstl 


By  adjusting  the  camera#  zoom  fcv-ei 
and  po^t'oo§.  h^  yw  hpve  a 

out  adjusted  you  can  arrange 
multiple  view^s  nr  the  pamcipanfc  rwm 
pretty  much  however  you  wa  m  For 
a  sample  rf  you  want  to  have  two 
views  oP  Ehe  partial  pan!  newt  In  a 
WC  window 


1  Set  the  Observer  Camera  Viewer 
■wmdow  lo  1 A1  Bize 

2  Use  lran^icn»9^lo-t5Qttom ). 
ar  a  PIP  window,  to  sptit  the  view 

between  the  cameras 

3  Hitfe-the  ngH  half  of  t  he 
C^mer^Vfew^r  window  t^rurbd  Ite 
VMC  window 

4  AdjU  Vt  the  cameras'  zoom  and  pa  n 
to  show  the  views  oP  the  participant 
that  you  want 


siicte  irw  t-Bar  to  irw  sop  peart  too  and  carnet  the 
j&ysJick  Sated  your  fa-o  inputs  [current  anct  ne*t),  by 
one  GUhgn  m  the  lop  rpw  pf  SOURCES  and 
one  in  the  bottom  row 


Side-by-side  cut 

Press  the  TRANSITIONS  -  [BASlC|  button 

Press  she  rtgfrt  arrow  to  move  the cursor  to  tradition 
n 7  (sda-by-side  cuts)  Press  OK 

Move  the  T'^ar  qp  and  down  3c  run  (he  bgrtsibun 
Adjust  the  cameras  if  simultaneous  side-by  side  views 
are  deeded  (with  the  T-8ar  half-way]| 


Picture-i n-pi dur#  (PIP) 

The  "neat'1  s&urce  wi  be  the  small  P\P  window 

Press  th*  pips  outten 

LT-se  the  T  -Ba  r  to  see  Ihe  PIP  window 

Use  the  joysStek  to  move  the  PsPvwidQw  into  position 


Press  [PLAY]  Id  start  ire  PIP  on  the  Piogram  output 

Be  careful  net  ic  burtc  toe  joystick  ek  T-bar  whino 
recorOngi 


To  comptelezy  rese  t  Ihe  mixer  ^including  firmware) 
back  Eb  factory  state  power  on  while  halting  down 
Shift  and  Setup  (manual  Ch13}  You  wHI  may  have  In 
*elup  the  input  [ROUTE]  agern 


Lsad  tirmyer*  v*  Etfumti 

iPAddress  ni.izo.i4T.  m  {gunc) 

SMM  255.255.Z5Z.0 

gateway  131.120.144.1 
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MAC  Address:  00-02-ea“4c-47-e6 

S/M:  050518 

Check  FOCUS  website  for  firmware  upgrades. 


miniDV  VCR 
(portable) 


Can  run  off  battery  or  A/C  power.  Needs  special  A/V 
input  cable  (3  RCA  F  to  1/8"  plug). 

Can  convert  analog  (RCA  or  S -Vi dec)  input  to  raw  DV 
over  firewire  for  capture  to  computer  -  useful  with 
laptop,  but  the  Canopus  MPEG-2  encoding  board  is 
preferred  when  available  (on  desktop  computer  only). 


Wireless  microphone  receiver 


Power  ON,  check  antenna 
Adjust  volume  as  required 


Mobile  device  camera  receiver  Power  on,  check  antenna 


Will  suffer  RF  interference  from  wireless  networking 
equipment 


Observer  microphone 


Push  To  Talk  (PTT)  -touch  and  hold  to  talk.  Can 
change  PTT  function  to  "always  on  "  or  other  settings, 
by  setting  DIP  switches  (see  mic  manual) 


Mic  amp  PreSonus  MP20  CHI:  Ceiling  mic  in  Participant  room 

CH2;  PTT  mic  in  Observer  room 
Both  channels: 

ON  (lit):  0:  +48V.  80Hz,  UR  OFF  (unlit):  PAD 
GAIN:  as  desired  IDS:  max  PAN:  center 


M  aste  r  Level :  as  desired  Phones :  as  desired 
Adjust  GAIN  to  control  a  mic's  recorded  volume. 
Speaker  amp  CHI:  Ceiling  speaker  in  Participant  room 

CH2;  Ceiling  speaker  in  Observer  room  -  note  that 
this  is  completely  separate  from  the  computer 
speakers,  which  may  (or  may  not)  be  playing  the 
same  audio 
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twadphones  OFF  (<m)  volume  about  6  on  eh  i  12 


Observer/Participanl  room  DnvEn  by  *rnp  (not  seif-powere^  Ensure 

Speakers  ran***™  aft  SMi 

N  ate  mat  mese  are-  Malty  separate  f  ram  She  computer 

speakers 


Woldus  HASP  USB  aerial  key 

LwMrtS*  number  O B050. 05422 -DGAA 


&e  vtty  cartful  w4h  it  this  t*  expensive  to 
if  left 

Do  NOT  update  dnvera  To*  the  Aladdin  USB  HASP 

tan  cause  your  USB  hasp  key  (cfongle)  andNUdut 

sohwar-e  So  slop  ^rfcing  Tq  conreci  it 

1  Download  the  lateel  Dortimand^ne  HASP  4  drivers 
from 

ftWww  Biaflflm  bo rn.ti.ij poo n as nvr.  j ■;: oAfe  nd i  &e r 

asp  and  save  mem  to  a  foHer  suefi  as  c  '£emp 

2  Pram  the  Windows  Start  meixi,  te-ieti  Run  and  then 
Syire  ciwd  {a  cpnrai^rd  window  win  open) 

3  Type  cd  ciftemp 

4  Remove  any  hflf<torare  key*  from  your  U5i  or  LPT 

ports 

5  Type  Hmstall  k  |fe  remove  exisUng  HASP  drrverfi) 

$  Type  hinslall  ti  ftp  install  dne  new  drivers} 

7  Close  she  window  and  ^sen  hasp  key 


Moldus  The  Observer 

License  number:  QE=Q50-S**£2-DGAA 

Regisfiernd  u4«a  with  ftokfus  Saiturj 
McKiy  Kachi  Tafitagi 

Updaisflip  v50  3Hrom 


5a!narid'in*Eallribafl  fey 

John  J  Me Orsw  SaaesEnamw  j  mcgraw^rieliius  com 

Sari  vjn  ftoefcel  bec^incal  rep  Ircm  the  Nethed  ands 
8  vanRoeswr^-Noldus  com 

No*dL»  I  flfof  malice  TecJinol-ggy  Inc. 

75i  Muter  Dnve 
SunW^E-S 

LeeUxjr-j  VA  Mi75 

Ptwe  TOH1- 0449800-355- 
Fa*  703.771-0*41 

ne^araK.  nermaiwscsang  nui^s  Pat^  Standard 

Tim*  IP ST}  tasaj  m-atto 

Hotdirs  Technical  Supcon  KiMd^H 
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tupptf l  r^quqHJ  lorm 

mo  trmM  *i  okt-ds  «mfa  iertfa:20MQj?&95 


Canopus  MPEG  Pro 
MVftimsX  PCI  card 


InflEfnal  PGI  card  -  hardware  MPEG-2  encoder/ 
decoder  Controlled  £y  the  Canopus  MedHaCrurse 
appicafion 


Ci  no  pu5  MedidGtufSG  Video  reconJer  software  tar  irre  Canopus  rard .  Used 

SO  encode  She  FPOgram  DUipUE  of  the  video  miwer  (and 
She  s=udio  inputs -itonnetted  directly  to  ihe  candj  to 
MPEGl  (smaiteO  or  MPEGS-  (recorwnendeti)  file* 

This  is  the  hghest  qualify  way  to  rtoortf  video  from 
She  video  cameras  Duty  tf  you  need  cofFTpL^er  screen 
stetfs  too,  you  must  Lr.se  Cam  las  ra  and  J#»  P/o&O 
Cafflerav^aiver  appfocated  f&  vttf&o  n^M 

WtoOCD  qua  Irty  •{  WPEG1 ,  M  PC  ft  to  AH  nun)  * 
atevei  IQMBftnm  (1M  **  650MB  =  t  CD) 

OVD  totality  (MPEG2,  M2P  Me  extension)  s  stout 

aOMBmm  (2  hr  -  4  9  G0  =  t-  DVD) 

The  Mtdraeru**  window  bypasses  the  Window* 
operating  system  and  carmot  be  captured  with  screen 
shots  or  Camtasia 

Before  recording  sea  fre  FORMAT  VCOMFEG1  t& 

She  smallest  files,  or  DVD  M  PEG2  for  hfgTrer  qua  Irfy 

B*tort  rtewding  stS  Sfte  Fill  to  record  to  (d£k  fetoter 

icon) 

can  also  »  used  to  ptey  mpeg-2  video. 


Uncompressed  OV  files  ftom  a  camcorder  (  avs 
ejflen^on)  vse  atootf  200  MB/  mtn  ■  13GB/  hr 


Canopus  MVtf  D^WPEG  Fr,te  Will  convert  a  raw  DV  Me  (usually  odwrtoaded  from  a 
Co/iuerter  camcorder j  to  MREG2  (or  M  pegi  )  Takes  almost 

f*aM  ime  to  process  (a  3Qmm  rile  lake*  stout  30rrwi  to 
convert) 


Elecard  2,0  MPEG-2  video  player  apptaadon  t  has  works  with 

HQWue  Observer  to  pipy  MPEG-2  «0ep  When  scoring 

videos  Wlthquihs  program  MPEG-2  playback  in 

NoWls  Observer  has  limited  fuvi bora lrty 
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internal  PCI  me  -  turns  anaie-0  wbn  input  rite  a 

digrtsl  &trearri  Does  nqt  encode  to  MPEG2,  bul 
(LMtlitee  trie  Canopus  card],  Uie  Picofci  Camera  Viewer 
window  can  De  screen  capture  Capita Eia.  or  the 
Windows  operating  Steffi 


The  Observer  Camera  We  wer 

Video  vifewy  appiiutoii  tor  im 
Pimio  PCr  card 


Lfsed  to  capture  program  output  of  Ihe  video  mwei 
w  ilh  Ca  mtafra  The  contents  0T  Uvs  wind  w  Can  be 
oaptuf&d  bv  scr«*i  sh«s  or  camiasia 


ChecK  Command  >  Live  To  start  vnJeo,  unghecfc 

Command  >  Live  lo  make  coniigurstion  charges 


Imps rSsint  SOttlngS- 

Cdflflg  >  Input  Configuration:  flits  .<r  vtd&Q 

lQOk$  ViWJJ 

J7  using  Rcoto  £-  Vxteo  rnpuT  /acfc  S- Vd  eo  via 

Mja.fei.4,fer,aa.BBai.  S:^jtea.,iaSiaaa 

tf  us.ng  ftcofo  BP*lC  mpui  /acrt  ComposHevra 
BMC  Composite  fiTSC/EiA 

Ccnflg  ^  Seating  and  Region  carl  Interest  Click 

either  1/1  oM.'?ir  Ecth  HpnwfUgl  pnd  V^ifrgl  r-.f-i - 

drag  window  to  desired  size 

Vie#  >  Always  On  Top  As  desired 
Vie#  >  Toolbar  Off 
View  >  Status  Bar  Off 


tefflntit  (««[  should  mi  «wee  &d|y&tmen*s?. 

Ccnflg  >  Colo  r  Format  R£ffi24 

Conf  Ig  >  Adjustments  Restore  defaull  values 

AC|L£;  during  vitteti  play  ilrve}  rf  desired 

Ccnflg  >  Filters  and  Settings  A!l  OFF  excep  Full 
range  ADC 

Conflg  >  Acquisition  Setting*  Frame  (fieWt  are 

graboed  m  a ry  orr»0 

I/O  Operations  No  sellingfe  should  need  to  be 
^nged  «n  iNttrwrv 

I/O  Operations  >  I/O  Lines  4  buicons  on  lop 
chatted 
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I/O  Operations  >  Trigger  Config:  Rising  edge 
I/O  Operations  >  Arm-Acquisition:  unchecked 
I/O  Operations  >  Live-TriggerStop:  unchecked 


Sonic  My  DVD  Studio  Deiuxe  Used  to  burn  DVDs  that  are  viewable  in  any  DVD 

Suife  player,  with  simple  menus,  etc.  Has  a  very  simple 

video  editor  that  can  trim  and  combine  clips  and  add 
titles,  but  not  much  else.  Will  “fit”  (transcode)  video 
files  to  the  available  size  on  the  DVD  disk.  This  is  not 
the  "LE"  version  that  comes  with  the  DVD  burner, 
which  can  not  do  editing,  resizing,  etc 


Techs mith  Camtasia  Recorder 
2.0.3 


Used  to  capture  (to  .AVI  video  file)  anything  shown  on 
the  computer  screen,  including  the  contents  of  the 
Camera  Viewer  window,  but  NOT  including  the 
Canopus  Mediacruise  window. 

Suggested  settings: 

View  menu:  Standard,  uncheck  all  toolbars 

Effects >  An  notation  uncheck  all  options 

Eff  e  cts  >  Au  d  i  o :  check  Reco  rd  Audio  (o  nly) 

Effects  >Cursor  Show  Cursor 

Effects>Zoom  &  Pan  uncheck  all  options 

Too  Is  >  0  pt  1  o  n  s  >  A  VI  ta  b  uncheck  Auto 

Configure  and  Time-lapse  capture.  Set  to  IS  O 
frames/sec  Check:  Interleave  audio  every  1  seconds. 


Tools>Options>AVI  tab  >  Video  Setup: 
Compressor:  DivX  codec,  [OK] 


Tools>Options>AV!  tab  >  Audio  Setup:  Format 

use  MPEG  Layer-3  if  available  (for  smaller  file  size), 
otherwise  PCM.  Attributes:  22  kHz  (or  more).  Mono. 
(16-bit,  if  PCM),  [OK] 

You  might  also  check  the  Windows  sound  settings: 


Start  Menu>Control  Panel>Sounds  and  Audio 
Devices > Audio  Tab,  Sound  recording  Default 

Device:  (Sound  card),  Volume  ensure  that  " Limed n" 
is  checked,  volume  to  max 


Tools>Opiions>File  tab:  check  Ask  for  File  Name 


Tools>Qptions>File  tab:  check  After  save,  play  the 
video,  uncheck  After  save,  edit  the  video. 
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Techsmith  Camtasia  Studio  A  simple  video  editing  application  that's  good  for  quick 
2  0  3  trimming  of  clips,  adding  sound  and  voice  annotation, 

for  exporting  to  Flash,  and  especially  for  adding  on¬ 
screen  text  and  comments. 


RealVNC  4.0  (viewer  mode)  Connect  to:  131 . 120 . 147 . 110  password:  noldus 

Right-click  title  bar  for  options. 

Useful  options:  view  only,  scale 


Event  Logging  Keyboard 
P.l.  Engineering  X-keys  SE 

PIE002-13  Jog  &  Shuttle 
Editor 


Used  to  code  events  in  IMoldus  Observer,  fTwd/rewind 
through  videos  in  Observer  (use  ObserverControLxk4 
file  as  a  template),  and  for  video  editing 


X-keys  Macro  Works  Sets  up  the  event  logging  keyboard  for  use  with 

different  programs. 

The  Noldus  Observer  basic  control  file,  to  use  as 
template  for  other  configurations,  has  settings  stored 
for  using  the  jog/shuttle  dial,  but  none  of  the  keys: 

ObserverControl *xk4 


DivX  codec  and  DivX  player 


Adobe  Premier  Pro  1.5 

Note  required  updates: 

Main  Concept  MPEG  encoding  1.1 
Windows  XP  KB886716 


A  highly  compressed  codec  for  AVI  files  that  is 
popular  for  movies  and  video. 

Download  free  from:  http://www . di vx . com 

The  codec  must  be  installed  on  any  computer  that 
needs  to  play  video  files  created  with  the  Observer 
system. 


High-end  professional  video  editing  program. 
Expensive  and  difficult  to  use.  It  can  do  multiple 
picture- in-picture  and  other  features  not  found  in 
cheaper  video  editing  programs,  but  is  not  easy  to  use 
for  simple  projects  If  you  can  take  the  time  to  learn  it, 
it's  the  most  powerful  video  editor  available  for 
Windows. 
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Participant  station 


Room  221a  Combination:  3,1,5 


Participant  workstation 
computer 


Computer  Name:  IT004151  (on  NPGS  domain) 


IP  address: 

SNM: 

gateway: 

DNS  servers: 

MAC  Address: 


131.120. 147 .110  (static) 
255 .255.252.0 

131 ■ 120. 144.1 

131. 120.18. 40 

131 . 120. 19.41 
GO- 04 -7 6- D7 -72- EC 


Administrator  account:  admin 

password:  HSILpa$$ 

Local  user  account:  participant 

password:  HSIL221a 


Windows  2000,  512MB  RAM,  2Ghz  Pentium  4  Two  18  GB 
drives 


VNC  Server  Allows  the  participant  computer  to  be  viewed  or  controlled 

by  the  observer  station.  Setup  VNC  from  the  Admin  account 
on  the  participant  workstation. 

Pan  /  tilt  /  zoom  cameras  Do  not  move  cameras  by  hand  -  only  by  remote  Press 
(2)  <HOME>  on  remote  if  moved  by  hand  (see  manual) 


Ceiling  microphone  Powered  by  mic  amp 


Wireless  microphone 

n 


Check  9V  battery,  have  spares  on  hand.  Turn  OFF  after 
use. 


Mobile  device  ca  mera  Low  battery  will  cause  interference  on  video 
Check  9V  battery,  have  spares  on  hand 
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Office  furniture 


Turn  OFF  after  use. 

Can  be  used  with  cable,  hard-wired  directly  to  an  RCA  video 
input  (for  higher  quality),  or  through  the  MDC  wireless 
receiver 


Check  arrangement  in  relation  to  cameras  and  microphone. 
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System  Hardware  Components 


One-way  mirrored  glass 
Observer  microphone 

Observer  workstation  monitor,  keyboard,  mouse 

Camera  joystick  control 

Video  mixer 

Event  logging  keyboard 


Program  BAA/  monitor 
Preview  BA N  monitor 
Wireless  mic  receiver 


Speaker  and  mic  amps 


Observer  workstation  computer 
Printer 


Camera  joystick  control  Video  mixer 

Workstation  mouse  Event  logging 
keyboard 


Mobile  Device  Camera  (MDC) 
attached  to  a  Ruggedized  PDA 


miniDV  camcorder  mounted  on  mini 
tripod 


0 

1  AW  E  Q  bservation  La  bo  rate  ry  Basic  Us  er  Guide 

Appendix  C.  Observation  Lab  Technical  Reference 

page  43 

131 


Wiring  Diagram:  VIDEO 
AWE  Center  Observation  Laboratory 


Participant 

Room 


Observer 

Room 


JOYSTICK 

CONTROLLER 


wireless  or  wired  connection 


ilo  avoid  RF  interference) 


S-VHS  VCR 
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Participant 

Room 


Wiring  Diagram:  AUDIO 
AWE  Center  Observation  Laboratory 
Observer 
Room 


II 


CHI  OUT  CH  2  OUT 


CH  1  IN  CH  2  IN 


WIRELESS  MIC 
RECEIVER 
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PI  COLO  car 


3V> 


Observer  workstation  (simplified) 
Video  mixer 


Speaker  amp 
Micamp 


Video  mixer 


Note  that  you  can  use  S-Video  either/or  RCA 
connectors  on  the  four  inputs,  and  must  select  it 
via  the  Route  menu  {MX-4  manual  ch  7P  pg  73) 


O 

AWE. 


AWE  Observation  Laboratory  Basic  User  Guide 
Appendix  C.  Observation  Lab  Technical  Reference 


page  46 


134 


Screen  S/iots  of  Typical  Configurations 

Useful  desktop  shortcuts  to  common  apps  (also  found  in  Start  Menu) 


Observer  running  full  screen  in 
background 

Camtasia  recorder  start/stop  window 
VNC  to  Participant  workstation 


Preview  of  Participant  room  in  Picolo 
Camera  Viewer  window 


in  ^prna.W  i  It  " 

Desktop  icons 

Picolo  Camera  Viewer  showing 
Picture-in-Picture  (PIP)  of  two 
cameras 

Canopus  recorder  controller 


Canopus  video  output  window  (cannot 
be  screen-captured) 

Canopus  recorder  controller  (partially 
hidden) 
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Appendix  D.  Portable  Observation  Laboratory 
Embarkation  List 

Computer  Case 

(1)  Laptop  computer 

(1 )  Laptop  computer  A/C  power  adapter 

(1)  USB  mouse 

(1)  1GB  USB  thumb  drive 

(2)  blank  CD-R  media 

(1)  DVD+RW  burner 

(1 )  DVD+RW  burner  power  adapter 

(1 )  DVD+RW  burner  USB  2.0  connecting  cable 

(1)  pad  of  writing  paper 

(3)  pens 
(3)  pencils 
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Mobile  Observer  Case  1  (wireless) 

(1)  Mobile  Device  Camera  (MDC) 

(1)  MDC  receiver 

(1 )  MDC  A/C  power  plug 

(1)  MDC  video  cable  (custom  plug  to  RCA  M) 

(1)  MDC  battery  power  supply 

(1 )  wireless  mic  transmitter 

(1 )  wireless  mic  lapel  microphone 

(1 )  wireless  mic  receiver 

(1 )  wireless  mic  A/C  power  plug 

(1)  wireless  mic  receiver  cable  (1/4"  phono  plug  to  dual  RCA  M) 

(1 )  portable  miniDV  VCR 
(1)  portable  miniDV  VCR 
(1)  portable  miniDV  VCR  battery 

(1)  portable  miniDV  VCR  A/V  input  cable  (3  RCA  F  to  1/8"  plug) 
(4)  spare  9V  batteries  for  MDC  and  wireless  mic 
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Mobile  Observer  Case  2  (camcorders) 

(2)  miniDV-firewire  camcorders 

(2)  miniDV-firewire  camcorder  batteries 

(2)  miniDV-firewire  camcorder  A/C  power  adapter 

(2)  miniDV-firewire  camcorder  memory  sticks  for  digital  camera  function 

(2)  mini  tripods 

(1)  Large  tripod  ? 

(2)  Voice  recorders 
(2)  External  mics 

(1)  spare  battery  for  miniDV  camcorders  and  miniDV  VCR 
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Mobile  Observer  Case  3  (accessories  and  supplies) 

(2)  surge  protectors 

(1 )  6'-1 0'  LAN  cable 

(1)  S-Video  cable 

(1 )  3-way  RCA  patch  cable 

(1)  Female-Female  RCA  connectors 

(1)  full-size  (6-pin)  firewire  cable 

(1)  mini  (4-pin)  firewire  cable 

(1)  full  size  USB  cable 

(1)  mini  USB  cable 

(1 0)  9V  batteries  for  MDC  and  wireless  mic 

(20)  AAA  batteries  for  voice  recorders 

(10)  miniDV  tapes 

(5)  blank  CD-R  media 

(5)  blank  DVD+RW  media 

(1)  set  headphones 

(1)  S-Video  to  RCA  adapters 

(2)  BNCto  RCA  adapters 
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Appendix  E:  Useful  signs  and  displays 


The  following  pages  contain  signs  and  displays  that  come  in  handy  around  the 
usability  lab. 


The  original  files  containing  these  signs  can  be  found  on  the  CD  accompanying 
this  user's  guide. 
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Camera  Controller  Presets:  Camera  1  (FRONT  OF  ROOM) 


1  -  DOOR 

2  -  DESK 

3 -TABLE 

4 -WHITE  BOARD 

5 

6 

Camera  Controller  Presets:  Camera  2  (SIDE  VIEW) 


1  -  DOOR 

2  -  DESK 

3 -TABLE 

4 -WHITE  BOARD 

5 

6 

0 
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NOTICE: 

Audio  and  Video 
Recording  Equipment 
may  be  IN  USE 
in  this  lab 
at  any  time 


RECORDING 

IN 

PROGRESS 

quiet,  please 


Participant  computer  work  station  instructions 


0 

AWE. 
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APPENDIX  C.  APPLIED  WARFIGHTER  ERGONOMICS  CENTER 

INFORMATIONAL  MATERIALS 


1 .  NAVAL  POSTGRADUATE  SCHOOL  PUBLIC  AFFAIRS  OFFICE 
ARTICLE 


New  Applied  Warfighter  Ergonomics  Center 
Measures  Human-Machine  Performance 

F riday,  F ebruary  1 1 , 2005  Article  by  Javier  Chagoya 


U  .S.  Marine  C  orps  C  apt  Koichi  T akagi  uses  the 
N  ol dus  Observation  System  in  the  new  NFS 
Applie  d  Warfi  ^iter  Ergonomics  Center  (AWE)  to 
gather  empirical  data  ah  out  the  interaction 
betweenhuman  performance  variables  and  a 
broad  range  of  tactical  equipment,  including 
computer  workstations,  handheld  deuces,  and 
weapons.  Participants  are  asked  to  perform 
specific  tasks  as  they  are  observed  through  a  one¬ 
way  mirror,  as  well  as  by  remotely- controlled 
distal  video  cameras^  microphones,  and  computer 
m  onitors  that  capture  operator  activity. 

Takag  uses  a  special  keypad  to  code  participants' 
behavior.  He' s  particularly  interested  in  aspects  of 
the  machine- hum  an  interface  considered  positive 
or  negative  by  users.  F  or  instance,  a  participant 
might  be  aggravated  by  a  poorly  designed  piece  of 
equipment,  or  pleased  by  a  specific  function. 

Taker's  vision  is  to  empirically  evaluate  human- 
system  performance  variables  related  to 
equipment  designed  for  warfighters,  and  make 
recommendations  to  designers  leading  to 
improved  system  performance  aid  warfighter 
safety. 

Phase  I  of  the  project  was  to  acquire  and  then 
m  aster  the  N  cl  dus  Observation  platform,  a 
collection  of  off- the-  shelf  components  and 
proprietary  software.  The  equipment  has  an  eye¬ 
tracking  system  that  records  points- of- gaze, 
identifies  triggers  of  interest,  and  measures 
physical  and  mental  workload  variables^  stress^ 
dexterity,  and  reaction  time. 

In  the  future,  Takagi  plans  to  apply  AWE’s 
capabilities  to  cognitive  task  analyses  using 
subject  matter  experts;,  and  to  record  focus  group 
and  after- action  report  sessions.  The  process  will 
also  evaluate  man-machine  interfaces  and 
m  easure  the  usability  of  hardware  and  software, 
including  training  effectiveness  and  retention. 
AWE' s  Phase  II  will  extend  the  system's  Phase  I 
capabilities  to  field  environments.  Some 
equipment  to  achieve  this  is  already  in  place:  a 
portable  digtal  camera,  which  attaches  to  a 
T acticom,  a  hardened  version  of  a  Blackbeny;  and 


Capt  Koichi  Takagi  sits  at  the  observation  console  where  he 
gathers  measurement  data  on  a  participant's  performance  in 
the  nert  rocm . 


a  portable  computer  system  for  performing  on  site 
analysis. 

After  demonstrating  AWE's  capabilities  to 
Professor^  Dave  N  etzer  and  Alex  B  or  detsky  F  eb . 
8,  the  decision  was  made  that  a  network  addition 
may  soon  follow.  This  will  allow  the  system  to 
transfer  data  seamlessly  in  the  field*  as  in  the 
laboratory. 

Phase  III  goals  are  to  build  an  anthropometries 
and  biomechanics  laboratory  to  evaluate  how 
military  equipment  impacts  warfi  ^iters'  ability  to 
perform  physical  tasks.  The  lab  will  include  a 
video-based  motion  capture  system  capable  of 
statically  and  dynamically  measuring  size, 
strength  kinematics,  range  of  motion,  gait,  and 
postural  sway  of  warfi^iter  subjects. 

Takagi  is  actively  looking  for  parti cipants,  and 
can  be  reached  at  the  HumanSystems  Integration 
Laboratory  ktakagi@nps.edu. 
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2.  APPLIED  WARFIGHTER  ERGONOMICS  OBSERVATION 
LABORATORY  DATA  SHEET 


Applied  Warfighter  Ergonomics 
Observation  Laboratory  Data  Sheet 


Technical  CapabjJEties 

□  c  jpuai re  behavior  in  die 
laboratory  Using  ittbolic 
Miners  mien muI  one¬ 
way  itiirror- 

□  C^apEuri:  MuV»r  m  the 
field  using  portable 
digilal  L^aJhcLii'JL^,  jiitd 

inobi  3e  iki  ice  camera  with  porlab  It  miniDV  XCR. 

□  Mia  and  record  audio  and  video  from  mu  Itiple  live  and 

recorded  |hdudm£  Direel  sero-cn-capluTe  of  a 

uscf*  CHTinpy tirr  Kciwn  *r  PI )A  kisoti)  h jiieli  XX I  MPF/W  \  I  PEfi4  com  pater  film.,  DVD,  and  \  ido4apc. 

□  radii  behavior  ,intl  other  *v«hfi  to  lirnu-xumpfll  \og%  ihuL  jru  itsjd  ta  Lhn  video  Pcrtann  rijonjus  slMinlitfll 

nnnh  4*  I  includm  c  inter-reler  reliability.  bj^i|iKiUI  iat  analyst*.  nnd  nonparanwitrie  slatutk-'i  3  on  the  tvxuIi*. 

□  Jump  Im  coded  oVenEs  on  video  for  arulysin  and,  creole 

vitikd-highUjibi  J  >  VDs  uf  result*  tor  sponunni  Mid 
deVe^pens 

■Research  Cap^brFrtres 

GDbssvc  juitidpantti)  while  u^mg  2  computer  or  other 
equipment  in  [he  lab. 

□  Record  com  ntaU-i  during  nn  a  tier- adion  repon  (AAR} 

dti-Ni  dt. 

□  Record  Elites  iewg  and  inkfi »  group  studio . 

□  ReOlsrd  panit  ipants  reaction*  to  |*LiJec1ed  videsh,  or  bo 

ICOHUim  r  ir-jj U  uVczf  Unudspc likcV . 

EaarTTpPcs  -of  passible  results  that  can  be  obtained 
"When  given  the  lank.  *hcntw\Tmrscff  Oi  f/pc  fruj/1".  7  tail  of  partieipaiiL*  £44lVl  performed  il 

corrally  -within  Ihc  allotted  6(5  neeond*  fFttteW  flnw  ry  cHfiqpfofr  CF.  "r,  rf  ftiSjJ.  5  participants  l.>  !*w> 
oOnriplnlcd  tlsL-  Iji  '  k  afltt  ruokdijj  hjOl;  ur  m^wiz  errors  £H.$  .i,  C  -  I$,2-Sh  and  J  p.nlteip.i.nl\  £25*14  Jid 
fifrf  complLlc  Ilk.  Usl:  Vi  illiLfi  5  mtnuLM  (gum?  tq>  ar  Gxpenenczd  lt  terfwtiLViJ  ttJaJjftjncrMnJ.  ” 

~EuU«u  1  and  2  on  the  In^edb^d  EX1  jA  wore  piowii  oocfcetly  on  Ite  fatrt  try  ttS**  of  tlio  limt,  button  3 
was  press  ul  coniscd}  on  the  IltsD  try  87"  *  of  Etie  time,  bul  button  -I  wax  pnssed  emrastly  on  I  lie  1  iml  try 
only  45%  of  the  time.  linnwn  On  hAlon  4  included  missing  Ltw  button  C  2^'  a  of  the  lime),  pressing  die 
vLTLinu:  hielkm  ( the  1  imek  anil  dropping  the  dftdoc  £9*™  nf  the  time)." 

'‘Dtirmj  disj  Aflcf  Ac  Ekrtt  Rcpsat  £  A  AR>  dL'-brid',  town  motiber  -?3  dp|icarcd  U>  ncsiw  E  lht  olhenv  jkc 
disagree  with  Btic  leu  1 11  Itddcfh  oommefUx  on  6 occasion!,  Nil  made  no  vcatul  c ommcul.  ACcmbcrs  3,2, 
and  ft  con  xisten  tly  rei  nEoteed  eaeh  other's  co oincnb  verbally  -nnd  ncm-i  erbally. " 

"llie  equspmenE  load  an  worn  inExifcred  with  ihu pvtidpanls1  ability  lo  shoulder thuic  weapons.  Et 
inereiwed  die  time  En  move  fitm  a  Minding  positkin  to  a  oOVHod .  hnuclin  g  lirinu  poi  rl ion  by  an  ai  erage  of 
.'  i  ^ecin^U  f  j?  ^j1  J  /'  fclWj.  1  lure  arc  some  fflaiinpk  %  kks>  clip* 

nd  Itic  jnu^tmenl  w-illL  ±nJ  ^tdiuuE  11il  carrying  load.  sEiuwiiig  lmw  it  jnfcrllcfud  “ 


TiM?  Cl 1  JVn-  SWfl- 
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Porfabfe  Observation  Labo-^to/y 

The  AWE  Center  observation  equipment  has  been  modified  at  me  Nav^i  Ross-graduate  Sobocu  tq  be  My 
portable  baffl  ary -powered  and  em  bailable  in  mi  I -standard  transit  cases  Our  research  capabilities.  incHuding 
multiple  dgrtal  audio  and  video  recording,  and  Full  computer  ana  lysis  capability,  can  be  performed  on-site,  afloat, 
in  the -field,  and  even  qn  tfie  move  In  menpack  cqnftgurgficsrrs 


Observer  wor  kstation  comp  uler 

(M&manmWftP*3.2Kta 

7QG&  HD.  1GB  RAM,  AfHflps- -Di©9GiGi  CO-RkMO  vO*R  W; 

ATI  Re&m  K300  Senes  1SQM0 

with  special  narCw  a  re  and  software  described  below 

Even  loflgi  rig  key  boa  rd 

FI  tnptneBMig  X-fcejs  EE 

Camera  joystick  control 

Tetemetncs  CP-ITV-  VC  4 

Video  ntoer 

F«u s  £ Jir?s^?@7tanfs  Wx-4 

■ninifrV  VCR  f  portable  | 

Sony  Cv-QTQOO 

Wireless  microphMiB  r^Mi  wr 

Atfunzm 

Mobile  device  camera  receiver 

Kerr?  Gig-asy  sran  ftX 

Observer  microphone 

SftuiB  \mi2Dfc 

Mkinp 

ProSorvLis  WP20 

Speaker  amj? 

0$  mr^nti  P*12 

Speakers 

Radio  Shack  400-4101 

BSlW  video  monitors 

Show  Eh*  preview  and  program  dtulpjis  orf the  >ndto  rmer 

One-way  m  i  rrore  d  glass 

4&  x  36' 

Observer  workstation  (special  hardware  a nef  software) 

Hbldus  HASP  U5-B  serial  kfry 

License  nurrOer  OB01d-0iJ22 -OSAA 

Must  be  present  to  use  Nofcdut  Observer  -software 

Holduj  The  ObfiOr rtr  vS.O 

Canopus  IMP  EC  Pro  MVR1000SX 
PCi  card 

Software  application  Tor  ceding  and  retorting  behavior 

Video  encoder  board  -  real-time  MPEG-2  encoder  from  analog  or  DV 
irput 

Canopus  Modiocruise- 

Program  to  encode  analog  Video  mpj  to  MPSC-2  files 

Canopus  N1VR  OV-MPEC  File 
Converter 

SOttwAFfc  to  ifihvfrfl  raw  DV  Fito*  16  MFEG-2  format 

Euresys  Ficolo  PCI  card 

Video  capture  board  (FCl  slot  2) 

Holdus  Camera  Vi  swef 

Viewer  for  video  in  to  PteoloCard  From  video  muxer 

Sonic  MyDUD  Del  axe  Suite  6.1 

Software  DVD  playerfrecorder  Watch,  edr:,  and  burn«  DVDs 

Techs.mMh  Camtssia  Studio  2-0-3 

Record.  edi?_  publish  videos  of  PC  desktop  actrwty 

ReaiVNC4.0  [in  viewer  mode] 

C4?«rv*  f  mUTQl  in?  pBrtipppntlf  wmpq^r 

X-keyb  Macro  Worhs 

Controls  the  event  legging  keyboard 

DivX  codec-  5.05,  DivX  player  2.6 

v«d» codec  and  player  For  highly  compressed  DhrK  y*dec  files 

Adobe  Premier  Pro  1.6 

Professional  video  erfiting  soFlware 

Participant  statran  eomponente 

Participant  workstation  computer 

Windows  2QOOj  51 2M  B  ^RAM  2Ghe  Pent]  usn  4  Two  1  fl  GB  drive  s 

CmJd  Be  a  ny  com  pUCer  but  must  run  RealVNC  Sew  er  rrequres 
admin.  aocwAto  iralalft 

Pan  /  tell  /  toom  cameras  |2) 

C aw  vC-C*  and  uC-CSOr 

Calling  microphone 

^nun?  MXSOQ  omnidirectional 

Wireless  microphone 

Azdfrn  Jf  D  r  Freq  A4  1 71  9C5  MbZ 

Mobil  e  device  camera 

rvdtrus  WDOMV 

OHice  furniture 

Individual  military  equipment 

GPS  Rebroadcaster 

Desk.  conference  tabte.  dry^rase  white  board,  dhairg 

Heime:  risk  pekei  molle  harness  and  cartridge  ben  rubber  m  -ies 
GFSStfuroe  GPSRKL1-P1  tOTi-NF  rransmitS  GPS  signal  indoors- 

TpJw^  els  Jvr4  2WI 
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APPENDIX  D.  POINTS  OF  CONTACT 


1 .  NAVAL  POSTGRADUATE  SCHOOL  (NPS) 


Dr.  Nita  Miller 

Director,  Human  Systems  Integration  program 


Dr.  Dave  Netzer,  Director,  CDTEMS 

Tactical  Network  Topology  (TNT)  program  director 


Marianna  Verett 

Dr.  Netzer's  research  assistant,  1ST  student 


Dr.  Alex  Bordetsky  (ITM) 
cell  831-521-9196 


Dr.  Eugene  Bourakov  (ITM) 


Dr.  Kevin  Jones 

Mech  Engr,  TNT  Aeronautics  expert 


Ryan  Hale 

Systems  Engineering  special  contract  research  assistant 


Francisco  Caceres 

Capt,  USMC.  ITM  Student  with  RPDA  knowledge  and  contacts 
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Maj  Carl  Oros,  USMC 
1ST  faculty 


Tom  Hyunh 
John  Osmundson 
Systems  Engineering 

LT  Lani  Bostick 

NPS  Supply  Officer,  Navy  Region  Southwest,  x2236 
(past  ballroom,  1st  deck  Hermann  Hall  by  comptroller) 


Matt  Simpson 

IstLt,  USAF.  HSI  Student. 


Derek  Read 

IstLt,  USAF.  HSI  Student. 


Koichi  Takagi 

Capt,  USMC.  OR  /  MBA  Student 
Naval  Postgraduate  School,  (OA  Code  61) 

L  k  HS  1411  Cunningham  Rd,  Rm  GL-219 
Monterey,  CA  93943-5221 

Chris  White 

Navy  Human  Performance  Center 
Jeff  Roth  a  I 

Reference  and  Instruction  Librarian  for  Operations  Research 

(831)656-2344 

jrothal@nps.edu 
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2.  RPDA  MANUFACTURERS 

Points  of  contact  to  the  specific  RPDA  manufacturers  is  available  in  a 
separate  file,  by  contacting  the  Operations  Research  department  at  the  Naval 
Postgraduate  School. 


3.  SPONSORS  AND  OTHER  AGENCIES 

United  States  Army  Special  Operations  Command 
(USASOC) 

Ft.  Bragg,  Fayetteville,  NC 
Brian  Kent  (LtCol  -  PM  at  JSOC,  handles  money) 
brian.l.kent@us.army.mil  (910)  243-0451,  cell  (910)  670-8976 
MAI  (Mobile  Access  Infrastructure) 

Joint  Tactical  C4I 

Kyle  D.  Longcrier 
Telecommunications  Specialist 
J-6  Technology,  JSOC 
Ph:  (910)243-0205 

Email:  kyle.longcrier@us.army.mil 
longcrierk@soc.smil.mil 

Rapid  Equipping  Force  (REF) 

Don  Taylor 

donald.taylorl  @belvoir.army.mil 
Project  Leader 

US  Army  Rapid  Equipping  Force  (Wexford  Group  Intnt'l) 

Civ:  (703)  704-2247 

DSN:  654-2247 

Cell:  (703)  200-7565 

or  cell:  (703)589-2150 

FAX:  (703)  704-4224 

http://www.ref.army.mil/ 
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4. 


LABORATORY  EQUIPMENT  SUPPLIERS 


Noldus  Information  Technology  Inc. 

751  Miller  Drive,  Suite-E-5 
Leesburg,  VA  20175 

Phone:  703-771-0440/800-355-9541 

Fax:  703-771-0441 

Web:  www.noldus.com 

John  J.  McGraw,  sales  engineer 
j.mcgraw@noldus.com 

Bart  van  Roekel,  installation  and  technical  Expert 
b.vanroekel@noldus.com 

Donna  Smith,  office  manager 
d.smith@noldus.com 

Potter’s  Electronics 

Michael  Barber 
1612  Fremont  Blvd 
Seaside,  CA 
393-0277 


GPS  Source  Technical  Sales  and  Support 

Richard  Reap 
719-561-9520 
sales@gpssource.com 
www.gpssource.com 
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5.  DEFENSE  LANGUAGE  INSTITUTE  (DLI)  AT  THE  PRESIDIO  OF 
MONTEREY  (POM) 


Col  Jeff  Cairns 

Garrison  Commander  (SOF  background) 
jeffrey.cairns@monterey.army.mil 

Marine  Corps  Detachment  (MarDet) 

Executive  officer  (XO) 

Capt  Raymond  Servano 
ravmond.servano@monterev.army.mil 

(831)242-5757 

MarDet  S-4/Supply 

Source  to  temp-loan  helmets,  flak  jackets,  packs,  782  gear 
242-6854 

Deck  Gunny 

POC  for  USMC  casuals 

SSgt  Boissoneau  nathan.boissoneau@monterey.army.mil 
242-5873 

S-3  Operations 

SSgt  Murray  keevin.murray@monterey.army.mil 


229th  Military  Intelligence  (Ml)  Bn 
S-3  (Operations) 

SFC  Swanson 
242-5295,  242-5208 

Source  to  temp-loan  military  equipment  and  coordinate  DLI 
Army  "casuals"  (students  not  currently  in  training).  “DSSOPS”  is 
in  charge  of  all  DLI  casuals. 

Chief  Hernandez  baltazar.hernandez@monterey.army.mil 
Person  to  send  Casual  Request  Form  to 

Williamson,  John  C.  (SSG) 

John.Williamson@monterey.army.mil 
Person  who  handles  casuals  for  229MI  Bn. 
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Bumann,  Gregory  (SSG) 

Gregory.bumann@monterey.army.mil 
242-5796 

NCO  in  charge  of  B  Co  casuals.  B  Co  may  have  better 
participants  and  availability  for  research. 

Drill  Sgt  Stroud 

Bldg  622  ("Bn  Mailroom"  etc.),  2nd  floor 
From  IMPS:  take  Del  Monte  to  Van  Buren,  left  on  Van  Buren, 
right  on  Franklin  St,  enter  DLI  Franklin  St.  gate,  take  first  left 
(Rifle  Range  Rd.),  enter  parking  lot  on  left. 


Petty  Officer  McGuire  david.mcguire@cnet.navy.mil 
PO  Buchanan  benjamin.buchanan@cnet.navy.mil 
242-7125 

In  charge  of  Navy  detachment  casuals  only.  Might  be  easier  to 
coordinate  directly  with  them  to  request  Navy-only  casuals. 
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APPENDIX  E.  SURVEYS 

TECHNOLOGY  EXPERIENCE  SURVEY 


Participant  ID  # 


Technology  Experience  Survey 

Instructions:  Please  circle  the  response  or  enter  an  answer  that  most  accurately  describes  your 
experience  with  the  following  technologies.  If  you  are  not  sure  about  an  answer,  make  your  best 
guess. 


Cell  Phone  Experience 


1.  Do  you  have  a  cell  phone?  (if  NO,  skip  to  question  7) 

[  YES  ]  [  NO  ] 

2.  Approximately  how  often  do  you  talk  on  your 
cell  phone? 

Please  enter  a  number 

(hours  per  week) 

3.  Approximately  how  often  do  you  use  text 
messaging  on  your  cell  phone? 

(hours  per  week) 

4.  Approximately  how  often  do  you  customize 
your  cell  phone  (change  ring  tones, 
background  pictures,  etc.)? 

(hours  per  week) 

5.  Approximately  how  often  do  you  play  games 
or  use  other  programs  on  your  cell  phone? 

(hours  per  week) 

6.  Approximately  how  many  phone  numbers  are 
stored  on  your  cell  phone  phonebook? 

Personal  Digital  Assistant  (PDA)  Experience 

PDAs  include  Palm  Pilots,  Pocket  PCs,  personal  organizers,  and  similar  devices. 


7.  I  use  a  PDA:  (Please  circle  one) 

1 

2 

3 

4 

never 

used  in  the  past, 

currently  use,  less 

currently  use,  30 

used 

but  not  now 

than  30  min  per  week 

min  per  week  or  more 

8.  My  experience  with  a  PDA  stylus:  (A  "stylus" 

is  a  small  pen  used  to  write  on  a  PDA  screen.) 

1 

2 

3 

4 

never 

used  in  the  past, 

used  in  the  past, 

used  in  the  past,  and 

used 

But  didn’t  like  it 

and  think  they’re  OK 

think  they’re  great 

Video  Game  Experience 


9.  How  many  hours  per  week  do  you  play  video  games? 

(hours  per  week) 

10.  I  play  online  video  games. 

[  YES  ]  [  NO  ] 

1 1 .  I  play  video  games  on  a  home  computer. 

[  YES  ]  [  NO  ] 

12.  I  play  video  games  on  a  video  game  system 
(Playstation,  X-Box,  etc.). 

[  YES  ]  [  NO  ] 

13.  If  you  play  video  games,  please  list  a  few  of  your  favorite  video  games. 

Technology  Experience  Survey 
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Participant  ID  # 


Computer  Experience  (Please  circle  one) 


14.  I  have  a  computer  at  work  that  is  primarily  for  my 
own  use. 

[  YES  ]  [  NO  ] 

15.1  have  a  computer  at  work  that  I  share  with  others. 

[  YES  ]  [  NO  ] 

16.  I  never  or  almost  never  use  a  computer  at  work. 

[  YES  ]  [  NO  ] 

17.  I  use  a  computer  in  my  living  quarters. 

[  YES  ]  [  NO  ] 

18.  I  use  public  computers  (at  a  library,  internet  cafe, 
base  facility,  etc.) 

[  YES  ]  [  NO  ] 

19.  I  often  use  several  programs  at  the  same  time. 

[  YES  ]  [  NO  ] 

20.  If  I  had  a  problem  while  using  a  computer,  I  would: 

Number  all  that  apply,  in  the  order  that  you  might  perform  them. 

a.  Stop  trying,  and  do  something  else. 

b.  Read  the  manuals. 

c.  Read  the  help  files  on  the  computer. 

d.  Ask  someone  for  help. 

e.  Keep  trying  different  ways  to  do  it. 

21.1  use  a  computer. . . 

Please  enter  an  approximate  number 

(hours  per  week) 

22.  I  send  and  receive  email  messages. . . 

(hours  per  week) 

23.  I  send  and  receive  email  attachments. . . 

(hours  per  week) 

24.  I  use  online  chat  or  instant  messaging. . . 

(hours  per  week) 

25.1  use  the  internet  to  find  information. . . 

(hours  per  week) 

26.  I  shop  or  browse  online. . . 

(hours  per  MONTH) 

27.  I  use  online  banking  services. . . 

(hours  per  MONTH) 

28.  I  trust  computers  to  store  sensitive  personal 
information. 

[  YES  ]  [  NO  ] 
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Participant  ID  # 


Military  Technology  Experience 

(expert  =  qualified  to  teach  the  subject) 


l=novice - ►  5=  expert 


29.  Military  radio  equipment. 

please  circle  one 

[N/A]  [1] 

[2] 

[3] 

[4] 

[5] 

30.  Military  GPS  (PLGRS). 

[N/A] 

[1] 

[2] 

[3] 

[4] 

[5] 

31.  Standard  topographical  map. 

[N/A] 

[1] 

[2] 

[3] 

[4] 

[5] 

32.  Standard  military  compass. 

[N/A] 

[1] 

[2] 

[3] 

[4] 

[5] 

33.  FBCB2  or  Blue  Force  Tracker. 

[N/A] 

[1] 

[2] 

[3] 

[4] 

[5] 

34.  Tacticomp  (indicate  when) 

[N/A] 

[1] 

[2] 

[3] 

[4] 

[5] 

35.  Commander's  Digital  Assistant  or  other  handheld 
computer  (please  describe). 

[N/A] 

[1] 

[2] 

[3] 

[4] 

[5] 

Demographic  Information 


36.  Age  (in  years) 

37.  Gender 

[MALE]  [FEMALE] 

38.  Pay  grade  (E-3,  0-2,  etc.) 

39.  Branch  of  Service  (USA,  USMC,  etc.) 

40.  Total  years  of  enlisted  service 

41.  Total  years  of  commissioned  service 

42.  Designator,  MOS,  or  rating 

(e.g  S 1/1/0,  0602  CommO,  11B  Infantry,  etc.) 

43.  Level  of  education  (please  check  one) 

_ 1.  less  than  high  school 

_ 2.  high  school  graduate  or  GED 

_ 3.  some  college 

_ 4.  college  graduate 

5.  graduate  degree 

44.  Current  duty  assignment  or  billet  description 
(e.g.  intelligence  analyst,  machine  gunner,  etc.) 

45.  Special  schools  and  qualifications  (circle  all  that  apply,  and  write  in  any  others) 

[a.  Ranger]  [  b.  Pathfinder  ]  [  c.  EIB  ]  [  d.  Air  Assault  ] 

[  e.  NCO  course  ]  [  f.  SNCO  course  ]  [  g.  Computer  Training  ]  [  h.  HALO  ] 

please  describe 

Other  courses: 
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HANDHELD  DEVICE  EVALUATION 


Participant  ID  # 


Handheld  Device  Evaluation 

Instructions:  Please  circle  the  response  or  write  an  answer  that  most  accurately  describes  your 
experiences  during  today’s  scenario.  Please  describe  your  experiences  only  during  today’s  scenario. 
Even  if  you  evaluated  a  particular  feature  during  a  previous  scenario,  describe  your  new  experience 
with  the  equipment  today.  Don't  fill  in  the  same  answer  as  before,  unless  your  opinion  is  exactly  the 
same. 

Use  additional  space  on  the  back  of  the  form  as  needed.  Your  detailed  additional  comments  are 
some  of  the  most  useful  results  we  obtain  in  this  study.  Please  be  frank  and  honest  -  all  responses 
will  be  kept  completely  anonymous. 


Scenario 


Communications  and  networking _ 

4.  Description  of  the  communications  and  networking  features  I  used  in  this  scenario. 


5.  Number  of  text  messages  sent,  (best  guess) 

_ a.  #  Attempted: _ b.  #  Successful: _ 

6.  Satisfaction  with  the  text  messages  I  sent:  ease  of  input,  speed  and  accuracy  of  entering  text, 
knowing  when  a  message  was  sent  and  received  by  the  recipient. 

very  somewhat  somewhat  very 

dissatisfied  dissatisfied  neutral  satisfied  satisfied 

[1]  [2]  [3]  [4]  [5]  [N/A] 

additional  comments: 


7.  Number  of  text  messages  received,  (best  guess) 

_ a.  #  Attempted: _ b.  #  Successful: _ 

8.  Satisfaction  with  the  text  messages  I  received:  ease  of  retrieving  and  reading,  knowing  when  a 
message  has  been  received,  accuracy  of  messages. 

very  somewhat  somewhat  very 

dissatisfied  dissatisfied  neutral  satisfied  satisfied 

[1]  [2]  [3]  [4]  [5]  [N/A] 

additional  comments: 
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Participant  ID  #  | _ 

9.  Satisfaction  with  the  communications  and  networking  functions  in  this  scenario. 

very  somewhat  somewhat  very 

dissatisfied  dissatisfied  neutral  satisfied  satisfied 

[1]  [2]  [3]  [4]  [5]  [N/A] 

additional  comments: 


10.  Satisfaction  the  reliability  of  the  communications  and  networking  functions  in  this  scenario. 

very  somewhat  somewhat  very 

dissatisfied  dissatisfied  neutral  satisfied  satisfied 

[11  [2]  [3]  [4]  [5]  [N/A] 

additional  comments: 


User  interface  -  software _ 

1 1.  Satisfaction  with  the  quality  of  still  images  displayed  (how  well  could  I  interpret  the  images 
displayed). 

very  somewhat  somewhat  very 

dissatisfied  dissatisfied  neutral  satisfied  satisfied 

[11  [2]  [3]  [4]  [5]  [N/A] 

additional  comments: 


12.  Number  of  moving  videos  I  viewed  in  this  scenario,  (best  guess) 

_ a.  #  Attempted: _ b.  #  Successful: _ 

13.  Satisfaction  with  the  quality  of  moving  video  displayed  (how  well  could  I  interpret  the  video 
displayed). 

very  somewhat  somewhat  very 

dissatisfied  dissatisfied  neutral  satisfied  satisfied 

[11  [2]  [3]  [4]  [5]  [N/A] 

additional  comments: 


14.  Number  of  audio  files  I  listened  to  in  this  scenario,  (best  guess) 

_ a.  #  Attempted: _ b.  #  Successful: _ 

15.  Satisfaction  with  the  quality  of  audio  files  (how  well  could  I  interpret  the  audio  presented). 

very  somewhat  somewhat  very 

dissatisfied  dissatisfied  neutral  satisfied  satisfied 

[11  [2]  [3]  [4]  [5]  [N/A] 

additional  comments: 
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16.  Number  of  times  I  used  the  mapping  functions  during  this  scenario,  (best  guess) 

_ a.  #  Attempted: _ b.  #  Successful: _ 

17.  Satisfaction  with  the  mapping  functionality  (how  well  could  I  interpret  the  map,  find  myself 
and  other  places  on  the  map,  how  it  compared  to  a  paper  map). 

very  somewhat  somewhat  very 

dissatisfied  dissatisfied  neutral  satisfied  satisfied 

[11  [2]  [3]  [4]  [5]  [N/A] 

additional  comments: 


18.  Satisfaction  with  the  situational  awareness  tools:  how  well  I  knew  where  myself,  and  friendly 
and  enemy  units  actually  were  (not  just  where  the  computer  said  they  were).  Also,  how  well  I 
knew  when  the  information  displayed  was  old  or  possibly  incorrect. 

very  somewhat  somewhat  very 

dissatisfied  dissatisfied  neutral  satisfied  satisfied 

Ml  [2]  [3]  [4]  [5]  [N/A] 

additional  comments: 


User  interface  -  hardware _ 

19.  Number  of  times  I  used  the  stylus  to  operate  the  device  during  this  scenario,  (best  guess) 

_ a.  #  Attempted: _ b.  #  Successful: _ 

20.  Satisfaction  with  using  the  stylus  during  this  scenario.  Please  comment  on  issues 
experienced. 

very  somewhat  somewhat  very 

dissatisfied  dissatisfied  neutral  satisfied  satisfied 

[11  [2]  [3]  [4]  [5]  [N/A] 

additional  comments: 


21.  Number  of  times  I  used  the  buttons  to  operate  the  device  during  this  scenario.  .  (best  guess) 

_ a.  #  Attempted: _ b.  #  Successful: _ 

22.  Satisfaction  with  using  the  buttons  on  the  device  during  this  scenario.  Please  comment  on 
issues  experienced. 

very  somewhat  somewhat  very 

dissatisfied  dissatisfied  neutral  satisfied  satisfied 

[11  [2]  [3]  [4]  [5]  [N/A] 

additional  comments: 
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23.  Satisfaction  with  one-handed  operation  of  the  device  during  this  scenario.  Please  comment 
on  issues  experienced. 

very  somewhat  somewhat  very 

dissatisfied  dissatisfied  neutral  satisfied  satisfied 

[1]  [2]  [3]  [4]  [5]  [N/A] 

additional  comments: 


24.  Satisfaction  with  removing  the  device  from  my  gear  or  carrying  position  to  use  it  during  this 
scenario. 

very  somewhat  somewhat  very 

dissatisfied  dissatisfied  neutral  satisfied  satisfied 

[1]  [2]  [3]  [4]  [5]  [N/A] 

additional  comments: 


25.  Satisfaction  with  stowing  the  device  on  my  gear  when  not  needed  during  this  scenario 

very  somewhat  somewhat  very 

dissatisfied  dissatisfied  neutral  satisfied  satisfied 

[1]  [2]  [3]  [4]  [5]  [N/A] 

additional  comments: 


26.  Ability  to  use  my  personal  weapon  while  operating  the  device. 

very  somewhat  somewhat  very 

dissatisfied  dissatisfied  neutral  satisfied  satisfied 

[1]  [2]  [3]  [4]  [5]  [N/A] 

additional  comments: 


27.  Ability  to  read  the  screen  under  various  conditions  (sun,  glare,  shadows,  night,  etc.) 

very  somewhat  somewhat  very 

dissatisfied  dissatisfied  neutral  satisfied  satisfied 

[1]  [2]  [3]  [4]  [5]  [N/A] 

additional  comments: 


Handheld  Device  Evaluation  page  4  of  5 

revised  08  Aug  2005 


164 


Participant  ID  # 

Overall  system _ 

28.  Satisfaction  with  the  reliability  of  the  system  to  perform  its  functions  in  this  scenario,  without 
malfunctioning,  crashing,  freezing,  incorrect  displays,  or  other  errors.  List  types  of  malfunctions 
experienced. 

very  somewhat  somewhat  very 

dissatisfied  dissatisfied  neutral  satisfied  satisfied 

[1]  [2]  [3]  [4]  [5]  [N/A] 

additional  comments: 


29.  Satisfaction  with  the  physical  ruggedness  of  the  system  to  perform  under  the  conditions  of 
the  scenario.  List  any  breakdowns  your  device  experienced. 

very  somewhat  somewhat  very 

dissatisfied  dissatisfied  neutral  satisfied  satisfied 

[1]  [2]  [3]  [4]  [5]  [N/A] 

additional  comments: 


30.  Overall  impression  of  the  system  during  this  scenario. 

very  somewhat  somewhat  very 

dissatisfied  dissatisfied  neutral  satisfied  satisfied 

[1]  [2]  [3]  [4]  [5]  [N/A] 

additional  comments : 


31.  Impression  of  how  easy  this  system  is  to  learn  and  use  for  me  and  people  like  me. 

very  somewhat  somewhat  very 

dissatisfied  dissatisfied  neutral  satisfied  satisfied 

[1]  [2]  [3]  [4]  [5]  [N/A] 

additional  comments: 


Free  comments 

32.  Please  take  as  much  room  as  you  require,  on  this  sheet  and  on  the  back  of  the  other  sheets, 
to  record  any  suggestions,  concerns,  and  random  thoughts  that  occurred  to  you  during  this  scenario. 
Thank  you  for  your  full  participation  in  this  survey. 
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APPENDIX  F.  USABILITY  TEST  SCENARIO 


1 .  INSTRUCTIONS  AND  SETUP 

Instructions  for  participants 

This  scenario  is  somewhat  artificial,  and  some  parts  may  not  make  sense 
tactically  or  realistically.  Do  your  best  to  keep  a  tactical  mindset  and  respond 
appropriately.  As  a  patrol  leader,  you  should  write  down  any  instructions  you 
receive  over  the  loudspeakers  (not  the  video),  because  you  may  need  to  brief 
your  patrol  later.  You  should  ignore  any  grid  coordinates  given  in  the  video. 

You  should  take  a  knee,  or  otherwise  react  to  the  scenario  as  you  might  in  a  real 
situation.  Please  stay  over  the  “X”  on  the  floor. 

Your  call  sign  is  “Alpha-Six",  or  just  "Six" 

You  are  talking  to  "Alpha  Company  HQ"  on  the  radio  and  on  the  RPDA. 

If  instructed  to  "tell  your  patrol"  something,  just  read  it  out  loud  in  a  clear  voice  as 
if  they  were  in  front  of  you. 

Tell  your  patrol  any  messages  you  receive  over  the  RPDA. 

Familiarize  yourself  with  the  following  points  on  the  map.  When  plotting,  use  the 
closest  location  on  the  map  that  makes  sense  to  you. 

Plot  routes  directly  from  point  to  point  -  do  not  try  to  stay  on  the  roads. 

Point  Pinos 
Bird  Rock 
Pescadero  Point 
Pebble  Beach  Quarry 

AlphaSix  Patrol  Base  (blue  rectangle  in  Pebble  Beach) 

Alpha  Company  HQ  (blue  rectangle  at  the  Presidio) 

Naval  Postgraduate  School 

Monterey  Peninsula  Airport  (use  center  of  runways  as  point  of  reference) 
Monterey  Commercial  Pier  (the  easternmost  pier) 

A  "QRF"  is  a  Quick  Reaction  Force. 


167 


Setup 

Plug  S-VIDEO  and  AUDIO  (headset)  cables  into  laptop. 

Set  up  laptop  for  dual  screen: 

Right  click  on  desktop  ->  Properties  ->  Settings  tab  -> 
click  monitor  2:  Resolution  800x600, 

Extend  my  Windows  desktop  on  this  monitor. 

Run  CDR  32:  Combined  Action  in  Iraq 
Screen  mode:  Dual-Screen 
Facilitator  notes:  Learn 
Language:  English 

To  play  a  video  in  the  CDR,  click  on  the  line  of  text,  not  the  icon. 
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2.  CONVENTIONAL  EQUIPMENT  SEGMENT  (Z) 

Tasks:  1c,  2c,  5c,  6c,  3c,  5c  (2nd  time),  6c  (2nd  time),  7c,  6c(3rc/  time),  4c 


(Z  -  Conv.  Equipment) 

Facilitator 

Participant 

Task 

Evaluated 

1.  As  a  group,  orient 
participants  to  the  important 
points  on  the  map  (listed  in 
Participant  Instructions). 

2.  As  a  group,  play  video  of 
RPDA  instruction  to  review 
functionality. 

3.  Allow  participants  10 
minutes  to  practice  on  RPDAs. 

4.  Answer  any  questions  about 
the  lab  environment  or  the 
RPDAs. 

5.  Have  participant  dress  into 
their  gear  and  calibrate  the 
RPDA.  Take  a  pencil,  notepad, 
protractor,  and  blank  map. 

Write  participant  number  on 
map  and  first  page  of  notepad. 

6.  Start  recording  on 
observer  computer. 

(Scenario 

setup) 

7.  Instruction:  “State  your 
participant  number.” 

States  participant  number  out 
loud. 

8.  Instruction:  "Break  out  your 
map  and  note  taking  gear  and 
stand  by  for  your  patrol  order. 
Disregard  the  coordinates  given 
in  the  video." 

Play  CDR  note  13.  (METT-T 
order) 

Play  title  screen  (to  take  the 
METT-T  order  off  of  screen) 

Watch  video 
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(Z  -  Conv.  Equipment) 

Facilitator 

Participant 

Task 

Evaluated 

9.  Orient  participant  to  map: 
Instruction:  "You  are  currently 
at  the  patrol  base  located  at 
grid  939506." 

Listen,  read  map 

10.  Instruction:  “Send  a  radio 
check  to  Alpha  Company  HQ" 

Turn  on  radio. 

Send  voice  comm  check. 

“Alpha  Company  HQ,  Radio 

1c  comm  chk 

Radio:  “Roger,  over” 

check,  over” 

“Roger,  Alpha-Six,  out” 

1 1 .  Instruction:  “Plot  a  direct 
patrol  route  with  two  legs  from 
the  Patrol  Base, 
to  Cypress  Point, 
ending  at  Pescadero  Point. 

Give  us  a  thumbs  up  when  you 
are  done.” 

Plots  route  on  map 

2c  plot  rte 
take  part¬ 
icipant  off  and 
on  task  as  he 
reads  info  or 
receives 
instructions 

12.  Instruction:  Tell  your  patrol 
the  six  digit  grid  coordinates  of 
each  point.” 

Reads  out  loud  three  grids: 

939  506 

916  487 

938  465 

13.  Instruction:  “Tell  your 
patrol  the  bearing  and  range  of 
the  first  leg,  and  the  total 
distance  of  the  trip.” 

Reads  out  loud  bearing  and 
range  and  total  distance. 

234°,  3100m 

Total:  6200m 

14.  Instruction:  “You  take  your 
patrol  on  the  road’ 

Watch  video. 

15.  Play  CDR  note  17.  (IED 
goes  off) 

Watch  video 

16.  Instruction:  “You  currently 
are  at  Bird  Rock.  Get  on  the 
radio  and  tell  Alpha  Company 
you’ve  been  ambushed.” 

Play  note  18.  (sporadic  fire) 

Radio:  “Alpha  company, 
we’ve  been  ambushed” 

5c  send  msg 

17.  Radio:  “Roger,  dismount 
and  engage  the  enemy.  What  is 
your  current  grid?” 

Reads  paper  map. 

Radio  “1  am  at  grid  925  499  “ 

6c  rev  msg 

3c  read  grid 
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(Z  -  Conv.  Equipment) 


Task 


Facilitator _ Participant _ Evaluated 


18.  Play  CDR  note  20.  (2 

Marines  down) 

Instruction:  “Request  an 
URGENT  MEDEVAC  FOR 

TWO  MARINES  WITH 
GUNSHOT  WOUNDS  AND 
HEAVY  BLEEDING” 

Radio:  REQUEST  AN 

URGENT  MEDEVAC  FOR 
TWO  MARINES  WITH 
GUNSHOT  WOUNDS  AND 
HEAVY  BLEEDING 

5c  send  msg 

19.  Radio:  “Roger,  medevac  on 
the  way.” 

Play  CDR  note  21.  (Helo 
Medevac) 

Watch  video. 

20.  Instruction:  “You  are  now 
at  Pescadero  Point.” 

Play  CDR  note  28  [Page  2]. 
(arrive  at  house) 

Watch  video. 

21 .  Radio:  "Alpha  Six,  what 
were  the  grid  coordinates  of 
your  original  patrol  route?" 

Radio:  Reads  back  grids  from 
notebook. 

939  506 

916  487 

938  465 

6c  rev  msg 

7c  read  back 

22.  Radio:  “Be  prepared  to 
displace  to  rally  point  at  the 
Pebble  Beach  Quarry  grid  958 
491” 

Radio:  “Roger” 

Copies  down  grid 

6c  rev  msg 

23.  Instruction:  “Tell  your 
patrol  the  bearing  and  range  to 
the  new  rally  point”. 

Uses  paper  map. 

Reads  out  loud  bearing  and 
range: 

36°,  3300m 

4c  brg  mg 

171 


3.  RPDA  SEGMENT  OF  SCENARIO:  RPDA-A 

Tasks:  la,  5a,  6a,  3a,  4a,  5a  (2?d  time),  6a  ("2nd  time),  7a,  4a  (2nd  time) 


(RPDA-A) 

Facilitator 

Participant 

Task 

Evaluated 

1 .  Instruction:  “Stow  your 
radio  and  map  and  deploy  the 
RPDA  at  this  time.  Announce 
out  loud  to  your  patrol  any 
instructions  received  on  the 
RPDA.  You  currently  are  at  the 
Naval  Postgraduate  School" 

Take  out  RPDA 

2.  Instruction:  “Send  comm 
check” 

RPDA:  “comm  check” 

la  comm,  chk 

3.  Play  CDR  note  30.  (enter 
house) 

Instruction:  “Send  a  message 
to  Alpha  Company  HQ  telling 
them  you  FOUND  SUSPECT 
WITH  WIFE,  SEVEN 
CHILDREN,  TEN  RELATIVES, 
ONE  AK-47’ 

Play  sporadic  gunfire 

Play  note  31  (Police) 
repeatedly 

RPDA:  ‘FOUND  SUSPECT 
WITH  WIFE,  SEVEN 
CHILDREN,  TEN  RELATIVES, 
ONE  AK-47’ 

5a  send  msg 

4.  RPDA:  “BRING  SUSPECT 
TO  HQ” 

Read  RPDA.  Say,  “BRING 
SUSPECT  TO  HQ”. 

6a  rev  msg 

R  Incfn  irtiniv  o  rr^\  i+q 

Plotq  rni itp  on  RPDA 

U .  II  IO 11  LI  vllUI  1 .  1  IUL  u  1  UU 

with  two  legs  from  your 
location  to  the  rally  point  and 
then  back  to  A  CO  HQ  at 
”  task  omitted 

nuio  1  UUIC  Ul  1  itFltm 

task  omitted 

2a  plot  rte 

Instruction:  “The  new 
checkpoint  is  at  the 
intersection  of  Hwy  68  and 

Route  1 .  The  Alpha  Co  XO  is 
at  the  commercial  pier. 

task  omitted, 
not  supported 
by  RPDA 
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(RPDA-A) 

Facilitator 

Participant 

Task 

Evaluated 

6.  Instruction:  “Tell  your 
patrol  the  6-digit  grid 
coordinates  of  each  point” 
note  that  RPDA-A  was  not 
calibrated  to  the  correct  grid 
coordinates  on  a  paper  map. 

Reads  out  three  loud  grids 
from  RPDA: 

A6. 

Checkpoint. 

A  Co  XO  at  commercial  pier. 
(acceptable  if  the  participant 
taps  in  the  right  vicinity  and 
reads  grid  correctly) 

3a  read  grid 

7.  Instruction:  “Tell  your 
patrol  the  bearing  and  range  to 
the  checkpoint,  and  the  total 
distance  of  the  trip 

Reads  out  loud  bearing  and 
range,  and  total  distance 
(not  supported  by  RPDA) 

121°,  566m 

4a  brg  rng 

8.  Play  CDR  note  33.  (angry 
mob) 

Instruction:  “Send  a  message 
to  Alpha  Company  telling  them 
‘ANGRY  MOB  BLOCKING 
WITHDRAWAL’ 

RPDA:  “ANGRY  MOB 
BLOCKING  WITHDRAWAL” 

5a  send  msg 

9.  RPDA:  “ROGER.  QRF 
ENROUTE” 

Play  CDR  note  35.  (QRF 
arrives) 

Say,  “QRF  Enroute”. 

Watch  video 

6a  rev  msg 

10.  Instruction:  “The  Alpha 
Company  commander  has 
arrived  with  the  Quick 

Reaction  Force.  He  asks  you 
to  read  out  loud  the  message 
that  you  previously  sent  on 
your  RPDA  about  what  you 
found  at  the  house. 

Go  through  message  log  and 
read  back  message. 

7a  read  back 
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(RPDA-A) 

Facilitator 

Participant 

Task 

Evaluated 

1 1  .The  A  Co  Commander 
instructs  you  to  proceed 
directly  to  Monterey  Airport, 
rather  than  go  through  the  rally 
point.  Tell  your  patrol  the 
bearing  and  range  from  your 
location  straight  to  the 

Monterey  Airport." 

Reads  bearing  and  range  off  of 
PDA: 

113°,  2000m 

4a  brg  rng 

12.  Play  note  36. 

Instruction:  “The  QRF 
subdues  crowd  and  you  return 
to  base.  The  mission  is  a 
success.  Stow  your  RPDA  and 
your  gear  at  this  time.  END  OF 
MISSION." 

Stop  video  recording. 
Instruction:  "Please  return 
your  pencil,  protractor,  map, 
and  notepad  to  the  table" 

Watch  video. 

Return  RPDA  to  table. 

Remove  gear  and  exit  room. 

13.  Give  participants  the 
Technology  Experience 

Survey  to  complete. 

Complete  survey,  dismissed. 
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4. 


RPDA  SEGMENT  OF  SCENARIO:  RPDA-B 


Tasks:  3a,  5a 


(RPDA-B) 

Facilitator 

Participant 

Task 

Evaluated 

1.  Instruction:  “Prepare  the 
RPDA  at  this  time.” 

Take  out  RPDA 

2.  Instruction:  “Find  the  grid 
coordinates  of  the  commercial 
pier,  the  Monterey  airport,  and 
the  Naval  Postgraduate 

School.” 

Reads  out  loud  grids  from 
RPDA: 

994  510 

027  498 

008  508 

(grids  may  be  considerably 
different  on  RPDA  if  the  map 
image  has  not  been  aligned) 

3a  read  grid 

3.  Instruction:  “Go  to  the 

Start  Menu  and  use  the  Text 
Chat  application.  Send  a  text 
message  requesting  an 
‘URGENT  MEDEVAC  FOR 

TWO  MARINES  WITH 
GUNSHOT  WOUNDS  AND 
HEAVY  BLEEDING’  ” 

RPDA:  REQUEST  URGENT 
MEDEVAC  FOR  TWO 
MARINES  WITH  GUNSHOT 
WOUNDS  AND  HEAVY 
BLEEDING 

5a  send  msg 
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5.  RPDA  SEGMENT  OF  SCENARIO:  RPDA-C 


(RPDA-C) 

Facilitator 

Participant 

Task 

Evaluated 

1 .  Instruction:  “Stow  your 
radio  and  map  and  deploy 
the  RPDA  at  this  time. 
Announce  out  loud  to  your 
patrol  any  instructions 
received  on  the  RPDA.  You 
currently  are  at  the  Naval 
Postgraduate  School" 

Take  out  RPDA 

2.  Instruction:  “Send  comm 
check” 

RPDA:  “comm  check” 

la  comm,  chk 

3.  Play  CDR  note  30.  (enter 
house) 

Instruction:  “Send  a  message 
to  Alpha  Company  HQ  telling 
them  you  FOUND  SUSPECT 
WITH  WIFE,  SEVEN 
CHILDREN,  TEN  RELATIVES, 
ONE  AK-47’ 

Play  sporadic  gunfire 

Play  note  31  (Police) 
repeatedly 

RPDA:  ‘FOUND  SUSPECT 
WITH  WIFE,  SEVEN 
CHILDREN,  TEN  RELATIVES, 
ONE  AK-47’ 

5a  send  msg 

4.  RPDA:  “BRING  SUSPECT 
TO  HQ” 

Read  RPDA.  Say,  “BRING 
SUSPECT  TO  HQ”. 

6a  rev  msg 

R  Inctri  mtinn  ■  o  rr^\  i+q 

Plntq  rm itp  on  RPDA 

U .  II  1 0 1 1  UwUUI  1 .  1  IUL  u  1  UU  tO 

with  two  legs  from  your 
location  to  the  rally  point  and 
then  back  to  A  CO  HQ  at 
”  task  omitted 

nuio  i  uuic  ii  r\run 

task  omitted 

2a  plot  rte 

Instruction:  “The  new 
checkpoint  is  at  the 
intersection  of  Hwy  68  and 

Route  1 .  The  Alpha  Co  XO  is 
at  the  commercial  pier. 

task  omitted, 
not  supported 
by  RPDA 
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(RPDA-C) 

Facilitator 

Participant 

Task 

Evaluated 

6.  Instruction:  “Tell  your 
patrol  the  6-digit  grid 
coordinates  of  each  point” 

Reads  out  loud  grids  from 

RPDA: 

938  465 

958  491 

974  513 

3a  read  grid 

7.  Instruction:  “Tell  your 
patrol  the  bearing  and  range  to 
the  checkpoint,  and  the  total 
distance  of  the  trip 

Reads  out  loud  bearing  and 
range,  and  total  distance 
(not  supported  by  RPDA) 

4a  brg  rng 

8.  Play  CDR  note  33.  (angry 
mob) 

Instruction:  “Send  a  message 
to  Alpha  Company  telling  them 
‘ANGRY  MOB  BLOCKING 
WITHDRAWAL’ 

RPDA:  “ANGRY  MOB 
BLOCKING  WITHDRAWAL” 

5a  send  msg 

9.  RPDA:  “ROGER.  QRF 
ENROUTE” 

Play  CDR  note  35.  (QRF 
arrives) 

Say,  “QRF  Enroute”. 

Watch  video 

6a  rev  msg 

10.  Instruction:  “The  Alpha 
Company  commander  has 
arrived  with  the  Quick 

Reaction  Force.  He  asks  you 
to  read  out  loud  the  message 
that  you  previously  sent  on 
your  RPDA  about  what  you 
found  at  the  house. 

Go  through  message  log  and 
read  back  message. 

7a  read  back 

1 1  .The  A  Co  Commander 
instructs  you  to  proceed 
directly  to  Monterey  Airport, 
rather  than  go  through  the  rally 
point.  Tell  your  patrol  the 
bearing  and  range  from  your 
location  straight  to  the 

Monterey  Airport." 

Reads  bearing  and  range  off  of 
PDA: 

73°,  9400m 

4a  brg  rng 
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(RPDA-C) 

Facilitator 

Participant 

Task 

Evaluated 

12.  Play  note  36. 

Watch  video. 

Instruction:  “The  QRF 

Return  RPDA  to  table. 

subdues  crowd  and  you  return 
to  base.  The  mission  is  a 
success.  Stow  your  RPDA  and 
your  gear  at  this  time.  END  OF 
MISSION." 

Stop  video  recording. 
Instruction:  "Please  return 
your  pencil,  protractor,  map, 
and  notepad  to  the  table" 

Remove  gear  and  exit  room. 

13.  Give  participants  the 
Technology  Experience 

Survey  to  complete. 

Complete  survey,  dismissed. 
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6.  RPDA  SEGMENT  OF  SCENARIO:  RPDA-D 


Tasks:  la,  5a,  6a,  2a,  3a,  4a,  5a  ("2nd  time),  6a  f2nd  time),  7a,  4a  ("2nd  time) 


(RPDA-D) 

Facilitator 

Participant 

Task 

Evaluated 

1 .  Instruction:  “Stow  your 
radio  and  map  and  deploy 
the  RPDA  at  this  time. 
Announce  out  loud  to  your 
patrol  any  instructions 
received  on  the  RPDA." 

Take  out  RPDA 

2.  Instruction:  “Send  comm 
check” 

RPDA:  “comm  check” 

la  comm,  chk 

3.  Play  note  30.  (enter  house) 
Instruction:  “Send  a  message 
to  Alpha  Company  HQ  telling 
them  you  FOUND  SUSPECT 
WITH  WIFE,  SEVEN 
CHILDREN,  TEN  RELATIVES, 
ONE  AK-47’ 

Play  sporadic  gunfire 

Play  note  31  (Police) 
repeatedly 

RPDA:  ‘FOUND  SUSPECT 
WITH  WIFE,  SEVEN 
CHILDREN,  TEN  RELATIVES, 
ONE  AK-47’ 

5a  send  msg 

4.  RPDA:  “BRING  SUSPECT 
TO  HQ” 

Read  RPDA.  Say,  “BRING 
SUSPECT  TO  HQ”. 

6a  rev  msg 

5.  Instruction:  “Plot  a  route 
with  two  legs  from  your 
location  to  the  rally  point  and 
then  back  to  A  CO  HQ  at  974 
513” 

Plots  route  on  RPDA 

2a  plot  rte 

6.  Instruction:  “Tell  your 
patrol  the  6-digit  grid 
coordinates  of  each  point” 

Reads  out  loud  grids  from 

RPDA: 

938  465 

958  491 

974  513 

3a  read  grid 
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(RPDA-D) 

Facilitator 

Participant 

Task 

Evaluated 

7.  Instruction:  “Tell  your 
patrol  the  bearing  and  range  of 
the  first  leg,  and  the  total 
distance  of  the  trip.” 

Reads  out  loud  bearing  and 
range,  and  total  distance: 

36°,  3300m 
total:  6400m 

4a  brg  rng 

8.  Play  note  33.  (angry  mob) 
Instruction:  “Send  a  message 
to  Alpha  Company  telling  them 
‘ANGRY  MOB  BLOCKING 
WITHDRAWAL’ 

RPDA:  “ANGRY  MOB 
BLOCKING  WITHDRAWAL” 

5a  send  msg 

9.  RPDA:  “ROGER.  QRF 
ENROUTE” 

Play  note  35.  (QRF  arrives) 

Say,  “QRF  Enroute”. 

Watch  video 

6a  rev  msg 

10.  Instruction:  “The  Alpha 
Company  commander  has 
arrived  with  the  Quick 

Reaction  Force.  He  asks  you 
to  read  out  loud  the  message 
about  what  you  found  at  the 
house,  that  you  previously  sent 
on  your  RPDA. 

Go  through  message  log  and 
read  back  message. 

7a  read  back 

1 1  .The  A  Co  Commander 
instructs  you  to  proceed 
directly  to  Monterey  Airport, 
rather  than  go  through  the  rally 
point.  Tell  your  patrol  the 
bearing  and  range  from  your 
location  straight  to  the 

Monterey  Airport." 

Reads  bearing  and  range  off  of 
PDA: 

73°,  9400m 

4a  brg  rng 
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(RPDA-D) 

Facilitator 

Participant 

Task 

Evaluated 

12.  Play  note  36. 

Watch  video. 

Instruction:  “The  QRF 

Return  RPDA  to  table. 

subdues  crowd  and  you  return 
to  base.  The  mission  is  a 
success.  Stow  your  RPDA  and 
your  gear  at  this  time.  END  OF 
MISSION." 

Stop  video  recording. 
Instruction:  "Please  return 
your  pencil,  protractor,  map, 
and  notepad  to  the  table" 

Remove  gear  and  exit  room. 

13.  Give  participants  the 
Technology  Experience 

Survey  to  complete. 

Complete  survey,  dismissed. 
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7.  TACTICAL  SCENARIO  MAP 
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APPENDIX  G.  RPDA  TRAINING 


1.  RPDA-A  LESSON  PLAN  BRIEF 


Advanced  Soldier  C2  Capabilities 
RPDA’s 


By:  Mr.  CyleFena 
Military  Advisor 


1SAUDUST2D05 

SD1  mi  2S1  if  u  inn/Dsos/ro^i 


Ex 


ADVANCED  SOLIDER  C2  CAPABILITIES 


PURPOSE:  The  purpose  of  this  training  is  to 
familiarize  DLI  Students  with  the  basic 
functionality  and  characteristics  of  the  RPDA-A 
software  with  an  emphasis  on  situational 
understanding,  voice  communications,  and  text 
messaging  through  a  multiplex  radio  RPDA- 
Architecture. 


IS  AUGUST2HJ5 

SD  im  12S 1  if  UUTWQ5B5/  D334 


Ex 


ADVANCED  SOLDIER  C2  CAPABILITIES 

ADVANCED  SOLDIER  C2  CAPABILITIES 

AGENDA 

RPDA-A  FAMILIARIZATION 

HISTORICAL  BACKGROUND 

TASK:  Become  familiar  with  basic  RPDA-A  software 

TRAINING  OBJECTIVES 

components. 

DESCRIPTION 

RPDA-AHITECTURE 

CONDITIONS:  Given  a  classroom  environment,  one 

SOFTWARE  FAMILIARIZATION 

primary  instructor,  all  components  of  the  RPDA-A  and 

SYSTEM  START  UP 

RPDA-B  systems  with  appropriate  media,  and  training 

RPDA-A  FUNCTIONS 

PRACTICAL  APPLICATION 

STANDARDS:  Become  familiar  with  the  software  and  all 

REVIEW/ AAR 

components  of  the  RPDA-A  and  TACICOMP  system 

RPDA-Ahitecture. 

isAUGUSTmre  Ijl/V/ 

sdi  mi  2si  if  u  UTD/[]ms/m3i 

15  AUG  UST2HJ5  Tj  Xj 

sd  im  12s  i  if  uuTTvmmr  m3 i 

ADVANCED  SOLDIER  C2  CAPABILITIES 
RPDA-A  FAMILIARIZATION 


There  are  five  basic  components  of  the  RPDA-A: 

Soldier  PDA 
Radio  (Short  Range) 

GPS /  Battery  Box  Adapter 
Headset 

Push  to  Talk  Button 


1SAUGUSTZ1D5 

SD1  mi  2S1  if  U  UTU/[J50S/tI334 


Ex 


ADVANCED  SOLDIER  C2  CAPABILITIES 


SOLDIER  RPDA 

>  Provides  core  capabilities 

-  Mapping 

-  Messaging 

-  Voice  over  IP 

>  Provides  extended  capabilities 

-  PackBot  Control 

-  Robotic  ATV  Control 

-  Weapons  Control 

-  Sensor  Monitoring 


IS  AUGUST2HJ5 
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ADVANCED  SOLDIER  C2  CAPABILITIES 


DIGITAL  RADIO 

•  802.1 1  b  WiFi  based  radio 

-  2.4GHz,  11  channels  (3  non-overlapping) 

•  Network  radio 

-  Multi-Hop  (>5) 

-  Bridging/Routing 

•  2.4— 0.6  mbps  max  throughput  for  1-4 
hops 

•  Internal  battery  with  ~60  minutes 
reserve 


RF  Range: 


Weight: 

Power: 

15  AUGUST  20  D5 


~400m  LOS  per  hop 
2  lbs 
3.2  W 


ADVANCED  SOLDIER  C2  CAPABILITIES 
GPS/  BATTERY  BOX 

>  Standard  military  battery 
with  enclosed  case 

-  BA5590  family. 

>  Cables  with  connectors  are 
permanently  attached  to 
the  battery  box  (‘flying 
leads’). 

>  12  Channel  WAAS  GPS. 

>  Power  conditioning. 

>  Headphone  jack  for  radio 
voice  communications. 


E* 
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ADVANCED  SOLDIER  C2  CAPABILITIES 


Assembly  and  Start-Up 


Ex 


ADVANCED  SOLDIER  C2  CAPABILITIES 
SYSTEM  ASSEMBLY  AND  START  UP 


TASK:  Place  the  RPDA-Ainto  Operation. 

CONDITIONS:  Given  a  classroom  environment,  one 
primary  and  several  assistant  instructors,  all  assembled 
components  of  the  RPDA-A  system  with  appropriate 
media,  and  training  aids. 

STANDARDS:  Successfully  assemble  and  initialize  the 
RPDArA  so  that  it  provides  Situational  Understanding, 
VOIP  Communications,  Robotic  Control,  and  Text 
Messaging. 


Ex 


ADVANCED  SOLDIER  C2  CAPABILITIES 
Components 


ADVANCED  SOLDIER  C2  CAPABILITIES 
ASSEMBLY 


CAUTION:  When  installing  cables  do  not  crimp,  force  or  pull  cables  All 
cables  are  designed  for  a  precision  fit 


RPDA-A  assembled  on  a  MOLLE 
configured  assault  pack 


1SAUGUST2IDS 
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RPDAA  assembled  on  a  MOLLE 
load-bearing  vest 

Ex 
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ADVANCED  SOLDIER  C2  CAPABILITIES 
ASSEMBLY 


RPDA-A  assembled  on  Body  Armor  RPDA-A  assembled  on  body  armor  in 

Afghanistan 


IS  AUGU3TZES 
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ADVANCED  SOLDIER  C2  CAPABILITIES 
ASSEMBLY 


Note:  Inspect  all  components  for  damage  prior  to  assembly. 


1. 

2. 

3. 


Insert  Battery  into  combination  GPS/  Battery  Box  Housing. 

Secure  butterfly  connectors  on  the  sides  of  the  Battery  Box  Housing. 
Insert  combination  PTT /  Headset  into  cable  labeled  "Audio." 


GP  S/  Battery  Box  Housing 


Audio/ Push  To  Talk 


IS  AUGUST2CDS 
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ADVANCED  SOLDIER  C2  CAPABILITIES 


ADVANCED  SOLDIER  C2  CAPABILITIES 


ASSEMBLY 


ASSEMBLY 


1. 

2. 


3. 


Insert  Primary  RPDA  Cable  into  RPDA;  place  thin  portion  of  cable 
through  Cable  Retainer  and  gently  pull  cable  rearward. 

Slide  cannon  plug  of  cable  forward,  with  the  slotted  portion  up,  until 
the  plug  is  firmly  seeded  into  the  RPDA. 

Rotate  the  plug  bezel  in  a  clockwise  motion  until  tightened. 


1 .  Connect  quick  Release  radio  plugs  to  GPS /  Battery  Box  Housing 
cable  labeled  "Radio." 


Quick  disconnect  for  radio  and  headset 


PDA  Cable  Assembly 
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ADVANCED  SOLDIER  C2  CAPABILITIES 
ASSEMBLY 

1  Place  two-way  spring  loaded  radio  power  toggle  in  the  "on"  position. 
(Toggle  should  be  pointed  toward  the  antenna). 


ADVANCED  SOLDIER  C2  CAPABILITIES 

TROUBLESHOOTING 

Loss  of  Communications 

PDA  Screen  Disappear 

1.  Check  all  Audio  cable  connect  ions 

1. 

In  the  event  the  RPDA-A  PDA 

to  include  PTT  buttons  to  insure 

screen  disappears  perform 

they  are  seeded  and  not  depressed 

the  following  steps. 

by  equipment. 

2. 

Ensure  the  RPDA.A  PDA  cable  is 
firmly  seeded  into  the  PDA. 

2.  Check  volume  settings  on  the  PDA 
and  in  the  “Symphone” 

3. 

Check  to  see  if  Radio  switch  is  in 

application. 

the  “ON”  position,  antenna  is 

3.  Check  “Symphone”  application  to 

connected  securely,  and  has 

insure  correct  “Talk  Group.” 

proper  orientation. 

4.  Determine  correct  orientation  of 

4. 

Ensure  battery  is  secure  and 

radio  antenna,  and  insure  radio 

correctly  inserted  into  GPS/ 

switch  is  in  the  “ON”  position. 

Batteiy  Box  Housing. 

5.  Go  to  “Tools”  menu,  select  “Find 

5. 

Press  “RESET”  and  re-enter 

Others”  and  list  soldiers  to 

correct  organizational  data,  and 
re-initialize  RPDA-A. 

determine  if  your  PDA  is  reporting 
to  the  network. 

IS  AUGUST2DDS 
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ADVANCED  SOLDIER  C2  CAPABILITIES 
TROUBLESHOOTING 


Loss  of  ICONS  or  GPS 

1.  Select  ‘Tools”  menu  and  select 
“GPS  Status”  to  determine  satellite 
connectivity. 

2.  Check  to  insure  the  GPS  node  on 
the  Battery  Box  Housing  is  not 
obstructed  by  equipment  or 
overhead  cover,  and  is  oriented  for 
best  connectivity. 

3.  Insure  Radio  cable  is  connected  to 
the  GPS  Battery  Box  Housing. 

4.  Turn  off  PDA,  Turn  off  radio, 
remove  battery  from  GPS/  Battery 
Box  Housing.  Check  battery  life. 

5.  Re  insert  battery  into  GPS/  Battery 
Box  Housing  and  secure,  turn 
radio  power  switch  to  “ON,”  and 
initiate  PDA  by  pushing  soft 
“RESET.” 


6.  Enter  correct  organizational  data 
and  initialize  RPDA4V  insure 
correct  mapping  image  is  posted. 

7.  Wait  for  GPS  to  acquire,  select 
‘Tools”  menu,  select  “Find  Others” 
and  “List  Soldiers”  to  insure  your 
radio  is  reporting  to  the  network. 


Ex 


PDA  Backlighting 

1 .  In  the  event  the  RPDAJ\  PDA 
screen  is  too  bright  or  too  dim 
perform  the  following  steps: 

2.  Select  “Settings”  from  the  main 
pull  down  menu  in  the  upper  left 
corner  of  the  PDA  screen. 

3.  Select  “System”  at  the  bottom  of 
the  PDA  screen,  and  choose  the 
“Backlight”  ICON. 

4.  With  your  stylus,  slide  the 
indicator  bar  to  the  desired  level 
of  lighting  and  select  “OK”  in  the 
upper  right  corner  of  the  PDA. 


IS  AUGUST  2]D5 
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Loss  of  Sound 

1.  Check  all  Audio  cable  connections  to 
include  PTT  buttons  to  insure  they 
are  seeded  and  not  depressed  by 
equipment. 

2.  Check  the  headset  connectors  for 
proper  connection. 

3.  Select  “Speaker”  ICON  in  the  top  of 
the  PDA  screen  and  determine  level 
of  volume  required. 

4.  Push  the  large  button  in  the  lower 
right  corner  of  the  PDA  and  the 
“Symphone”  application  will  appear. 

5.  With  the  sty lus,  slide  the  vo lu me 
control  to  the  right  to  increase 
volume,  to  the  left  to  decrease 


ADVANCED  SOLDIER  C2  CAPABILITIES 
TROUBLESHOOTING 


ADVANCED  SOLDIER  C2  CAPABILITIES 


START  UP  "RPDA-A” 


NOTE:  Press  soft  "RESET”  to  clear  RPDA-A  memory  prior  to  each  use. 
This  will  automatically  start  the  PDA 

1 .  Press  the  "POWER"  button  firmly  in  the  upper  right  comer  of  the  soldier 
PDA  for  normal  operations. 

2.  The  PDA  should  initiate  and  give  you  a  “Symphone"  screen. 

3.  With  your  stylus  touch  the  "X"  in  the  upper  right  corner  and  you  will  see 
the  "RPDA-A"  start-up  screen. 

4.  With  the  stylus,  touch  the  "Start"  ICON  in  the  upper  left  corner  of  the 
select  "RPDA-A"  to  initiate  the  RPDA-A 

RPDAA  Start  Menu 


ADVANCED  SOLDIER  C2  CAPABILITIES 
START  UP  “RPDA-B” 


1.  Power  the  RPDA-B  by  pressing  the  power  indicator  in  the  top  right-hand 
corner  of  the  RPDA 

2.  Close  the  “GPS  Network  Settings”  screen  by  single  clicking  the  X  on  the  upper 
right-hand  corner  of  the  screen. 

3.  On  the  desktop,  double  click  “  My  Computer.” 

4.  Double  click  “Storage.” 

5.  Double  click  “sql.” 

6.  You  will  nowsee  a  message  that  says  “Already  Installed.”  Chose  “OK.” 

7.  A  navigation  window  will  appear,  chose  “OK.”  This  will  complete  the  installation  of 
the  SQL  Server  CE. 

8.  Click  the  X  in  the  upper  right-hand  side  of  the  screen  to  close  my  computer. 

9.  Double  click  “My  Computer”  once  again  and  navigate  to  \Flash. 

10.  Double  click  “Symphone,”  then  double  click  Symphone  (a  small  icon  with  a 
lightening  bolt  through  it. 

11.  Once  Symphone  starts,  the  top  will  read  “acquiring  group ’’...wait  until  the  screen 
reads  “group  acquired.” 
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ADVANCED  SOLDIER  C2  CAPABILITIES 


START  UP  “RPDA-B” 

12.  Minimize  Symphone  by  single  clicking  555-000x  on  the  bottom  of  the  screen. 
You  will  nowsee  a  small  icon  lightening  bolt  in  the  system  tray  area. 

13.  Click  the  X  in  the  upper  right-hand  side  of  the  screen  to  close  my  computer. 

14.  Double  click  “My  Computer”  once  again  and  navigate  to  \  Flash. 

15.  Doubleclick  “RPDA-A”  then  double  click  RPDA-A  with  the  blue  icon  to  start  the 
application. 
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ADVANCED  SOLDIER  C2  CAPABILITIES 
STARTUP 

1 .  The  duty  position  and  unit  organization  screen  will  appear. 

2.  Enter  your  unit  and  individual  data  to  change  position/ task  organization 
and  tap  "OK"  using  the  stylus. 

3.  The  RPDA-A  application  will  begin  loading  and  give  you  the  status. 

4.  Once  the  application  is  loaded,  a  map  will  appear  with  your  current 
position  indicated  on  the  map  in  your  squad  color  with  the  appropriate 
weapons  symbol. 

5.  To  verify  your  position,  move  in  any  direction  to  insure  your  ICON 
is  in  the  correct  location. 
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ADVANCED  SOLDIER  C2  CAPABILITIES 


ADVANCED  SOLDIER  C2  CAPABILITIES 


MENU  NAVIGATION 


MENU  NAVIGATION 
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ADVANCED  SOLDIER  C2  CAPABILITIES 
ORGANIZATIONAL 

1.  Display  "List  of  Soldiers."  This  function  allows  you  to  see  the  task 
organization  of  your  unit  and  who  is  operating  on  your  network. 

2.  Select  “Tools,"  "View Tree and  specific  soldier;  the  map  will 
"Auto  Center"  on  that  soldier. 

3.  Select  “Tools"  and  "Find  Others"  and  the  display  will  show  who 
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ADVANCED  SOLDIER  C2  CAPABILITIES 


Mapping  Data 
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ADVANCED  SOLDIER  C2  CAPABILITIES 


MAPPING  DATA 

1.  GPS  LEGEND 

UNIT_ COLOR 

SELF  RED 

1st  SQD  PURPLE 

2^SQD  LT.  RED 

3rd  SQD  GREEN 

ROBOTS  YELLOW 

BAD  GPS  GRAY 

REPORTING 


2.  Scale  Bar  at  the  top  is  graduated  in  meters. 

3.  Ten  digit  MGRS  coordinate  is  at  the  top  of  the  screen. 

4.  Zoom  "In"  and  "Out"  at  the  bottom  of  the  screen. 

5.  North  is  always  the  top  of  the  PDA  so  orient  yourself  accordingly. 

6.  The  GPS  updates  every  5  seconds. 
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ADVANCED  SOLDIER  C2  CAPABILITIES 
ICON  FUNCTIONS  AND  OPTIONS 


ADVANCED  SOLDIER  C2  CAPABILITIES 
ICON  FUNCTIONS  AND  OPTIONS 


IP  Address: 

123456789 

Latitude:  33 .70041 

Longitude: 

112.091786 

Status: 

OnlineOnGPS 


1SAUGUST3DS 
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With  stylus,  touch  “Send  Msg”  (Message)  and 
the  text  messaging  program  will  appear 
allowing  the  user  to  send  a  text  message 
without  having  to  navigate  back  through  the 
main  menu. 

With  the  stylus  touch  “P2P  Call”  and  the  RPDA-A 
will  allow  the  user  to  initiate  a  point  to  point 
call.  To  end  the  call  simply  touch  “P2P  Call” 
again. 

With  the  stylus  touch  “Show  Details”  and  the 
RPDA-A  will  show  name,  rank,  duty  position, 
location,  IP  address,  and  status  of  GPS,  and 
digital  radio. 
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ADVANCED  SOLDIER  C2  CAPABILITIES 


MAPPING  DATA 

7.  Select  "Map"  and  "GPS  Status;"  this  screen  will  give  you  time 
since  last  GPS  update,  number  of  satellites  currently  tracking 
your  position,  ground  speed,  elevation,  ZULU  time,  position  in 
MGRS,  and  position  in  LAT /  LOG. 

MAP  CALIBRATION 

1  Find  a  well  defined  location  on  the  map  and  preposition  yourself  there. 

2.  Refine  your  position  by  using  the  "Zoom"  bar  at  the  bottom  of  the 
Screen. 

3.  Select  "Map"  and  select  "Start  Calibration"  and  you  will  be  prompted 
with  instructions. 

4.  Follow  the  instructions  step-by-step. 
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ADVANCED  SOLDIER  C2  CAPABILITIES 
MAP  CALIBRATION 

5.  With  your  stylus,  touch  the  designated  location  on  the  map. 

NOTE:  Be  very  precise  in  designating  your  new  location . 

6.  The  RPDA-A  will  prompt  you  to  restart  the  system. 

7.  Select  "Menu”  at  the  bottom  of  the  screen,  and  select  "Exit." 

8.  Select  the  "Start"  ICON  at  the  top  of  the  PDA,  select  "RPDA-A”  to  restart 
the  system.  Once  restarted  follow  the  start  procedures  and  continue 
operations. 


ADVANCED  SOLDIER  C2  CAPABILITIES 


Text  Messaging 
and 

Ancillary  Features 
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ADVANCED  SOLDIER  C2  CAPABILITIES 

TEXT  MESSAGING 

Pre- Formatted  Messaging 

Text  Messaging 

r  _  "  a 

Incoming/  Outgoing  Mail 

Outgoing 

1  Select  "Menu”  and  "Outgoi  ng " 

2  Select  "Keyboard"  and  begin  typing  message 

Un-select  keyboard  when  typing  is  completed. 

3.  Select the TO’  pulldown  menu  and  you  will 

be  prompted  with  active  addressee 

1'  Ha  - 

4.  Select  address  of  the  intended  recipient  and 

seled  "SentT  at  the  bottom  of  your  screen 

triccmfna 

1  Select  "Menu"  and  "!ncoming " 

s 

2.  Select  the  specific  message  and  read. 

3.  To  reply  select  "Reply'  or  "Forward." 

nrucusrxxB 

ssiifflii&truimwms* 
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ADVANCED  SOLDIER  C2  CAPABILITIES 
POCKET TERP  FEATURES 

*  Select  Menu  and  Internet  Explorer. 

*  Select  language  (Pashtu  or  Arabic). 

-  With  the  "Stylus"  tap  Vitlage  Greetings  (a  list  of  standard  phrases 
should  appear). 

■  Select  a  specific  phrase 

*  Tap  'PLAY"  with  the  Stylus  to  listen  to  the  language  translation 

*  Tap  blue  "Back  Arrow/'  at  bottom  of  screen  to  return  to  previous 

let). 


*  Tap  Home  [COW  to  go  to  Sokfier  Information 

",  v  •  •  - 

soimu 
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ADVANCED  SOLDIER  C2  CAPABILITIES 

SUMMARY 

HISTORICAL  BACKGROUND 
TRAINING  OBJECTIVES 
DESCRIPTION 
RPDA-AHITECTURE 
SOFTWARE  FAMILIARIZATION 
SYSTEM  START  UP 
RPDA-A  FUNCTIONS 
PRACTICAL  APPLICATION 
REVIEW/  AAR 

stffl*  TPX 
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QUESTIONS? 
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2.  RPDA-B  STARTUP  INSTRUCTIONS 

1 .  Press  power  button. 

2.  "GPS  Network  Settings"  -  Tap  [OK], 

3.  Tap  [Stop  GPS]. 

4.  (Double  tap  to  open/activate  icons,  like  a  double-click  in  Windows): 
Calibrate  >  [Calibrate]  >  ignore  arrow  on  screen,  line  up  stylus  tip  with 
target. 

5.  Repeat  Step  4  until  satisfied,  close  Calibrate  app. 

6.  My  Computer  >  Flash  (folder)  >  ARC  (folder)  >  ARC  (application). 

7.  Click  [Ok]  on  the  three  (or  more)  error  message  boxes. 

8.  Use  ARC  program  as  with  RPDA-A:  set  Manual  GPS. 

9.  NOTE:  Using  Menu  >  Mail  to  chat  will  crash  the  device. 

10.  For  Chat:  Start  Menu  >  Text  Chat. 

1 1  .Tap  in  text  area,  call  up  keyboard  (icon  in  lower  right).  Tap  send. 

12.  Close  Chat  window  when  done. 

13.  To  shut  down  or  restart  RPDA,  use  Start  Menu  >  Suspend. 
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3. 


RPDA-C  LESSON  PLAN  BRIEF 

Note:  additional  teaching  notes  can  be  found  in  the  “presenter’s  notes”  in 
the  original  Powerpoint  files. 


Basic 
RPDA 
Functions 
(RPDA  C) 


O 

AWE, 


Lesson  Plan 


*  Overview  of  device  C 

*  Messaging 

-  Send  message 

-  Receive  message 

-  Recall  old  messages 


O 

AWE 


Instructor 


•  Mapping 

-  Locate  own  position 

-  Determine  bearing  and  range  to  point 

-  Orient  on  map  features 


Overview 


O 

AWE 


Messaging 


-  Only  a  small  portion  of  full  features  will  be  covered 

•  Power  On,  Off  (leave  ON) 

>  Screen,  stylus 

-  Tap.  double-tap  (stylus) 

•  Short  push,  long  push  (buttons) 

•  Virtual  Keyboard 

•  Main  applications:  ArcPad  and  Text  Chat 

-  ArcPad:  Short  push  on  “4”  button  to  launch 

-  Text  Chat:  Long  push  on  “2”  button  to  launch 

•  Look  for  “X”  button  in  upper  right  corner  to  close 
windows 

•  It  will  not  be  necessary  to  use  other  applications  on  the 
RPDA 


Hold  down  “2"  button  until  Text  Chat  window 
appears 

-  If  "T  button  is  pushed  too  briefly,  a  different 
application  will  open.  Close  this  application  using  the 


X1 


Launch  virtual  keyboard  with  a  firm  tap  on  the 
stylus  icon  (2nd  from  right  in  bottom  taskbar) 

Tap  in  the  white  area  of  the  Text  Chat  window  to 
see  the  blinking  cursor  and  then  use  virtual 
keyboard  to  enter  message  by  tapping  firmly  on 
letters 


•  Tap  Send 
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o 

AWE 


Messaging 


O 


Mapping 


-  Scroll  through  gray  area  in  Text  Chat  window  to 
read  previous  messages  (both  sent  and 
received) 

*  Can  scroll  and  move  windows 

•  DO  NOT  hit  the  “X”  box  on  the  Text  Chat 
window  when  switching  to  another  application; 
leave  it  open.  Doing  otherwise  will  erase  your 
message  log  and  undermine  your  ability  to 
receive  messages 


-  Short  Push  on  the  “4n  button  will  start 
Arc  Pad  application 
*  Relevant  Menus  are: 

-  2nd  Row,  4th  from  left:  Measure,  Advanced 
Select  (bearing,  range,  and  grid  coordinates) 
-2nd  Row,  1st  on  left:  Zoom  In,  Zoom  Out,  Pan 


•  To  close  the  virtual  keyboard,  firmly  tap  on  the 
keyboard  icon  (2nd  from  right  in  bottom  taskbar) 


0 

AWE 


Grid  Coordinates 


0 

awe 


Bearing  and  Range 


*  Advanced  Select  (6th  in  appropriate  pull 
down  menu) 

*  Tap  on  the  desired  point 

•  In  the  resulting  Feature  Information  box, 
tap  the  “+”  next  to  XY  Coordinates. 

•  Scroll  down  to  the  bottom  for  MGRS 
coordinates. 


-  Select  Measure  from  the  appropriate  puff 
down  menu  (2nd). 

■  Tap  on  each  of  the  two  points  using  the 
presented  ruler  icon.  This  should  create  a 
solid  red  line  between  the  two  points. 

•  Tap  on  Measure  again  (in  the  taskbar)  to 
bring  up  the  Measurement  Information 
window  and  find  the  bearing  and  range 
(From  ->  To  and  Distance)  of  the  trip. 
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Zooming  and  Panning 


© 
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Practical  Application  - 
Messaging 


*  Be  sure  to  select  the  "Pan"  feature  before 
trying  to  scroll  around  or  tap  on  map 


-  Start  Text  Chat,  open  Arcpad,  return  to 
Text  Chat 


*  To  zoom  in  or  out,  select  the  appropriate 
feature  and  tap  on  the  desired  center  of 
the  zoomed  image 


*  Send  a  comm  check 

*  Send  three  and  receive  three  text 
messages 


*  Be  patient  when  using  these  applications 


*  Recall  the  first  message  you  sent  (using 
the  message  log). 
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Practical  Application  - 
Mapping 


*  Open/Return  to  ArcRad 

*  Find  the  grid  coordinate  of  a  specific  point. 

*  Use  zoom  and  pan  to  find  various  points 


•  Find  the  bearing  and  range  from  one  point 
to  another. 
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4. 


RPDA-D  LESSON  PLAN  BRIEF 


['yY . |  Lesson  Plan 

Overview 

*  Overview  of  device  D 

*  Messaging 

-  Send  message 

-  Receive  message 

-  Recall  old  messages 

*  Mapping 

—  Locate  own  position 

-Determine  bearing  and  range  to  point 

-  Orient  on  map  features 

■  Only  a  small  portion  of  full  features  wilt  be  covered 

-  Power  On,  Off  (leave  ON) 

-  Screen,  stylus 

-  Tap,  tap  and  hold,  double-tap 

-  Virtual  Keyboard 

-  Ma  in  a  pp  lication :  C2GE 

-  Slurt->  CJCfc  (il  necessary) 

*  Look  for  “OK"  button  in  upper  right  corner  to  close 
windows 

*  DO  NOT  attempt  to  use  other  applications  on  the  RPDA 

Messaging 

Message  Log 

-  Start  in  C2CE  Main  Window 

•  Tap  handset  icon  (4tn  from  left) 

•  Tap  radio  with  green  check  for  “Comm  Check” 

•  Tap  in  Composer  area  (so  that  cursor  blinks 
there) 

•  Use  virtual  keyboard  to  enter  message 

•  Tap  Send  All  (envelope  All)  icon 

•  Scroll  through  Communications  Log  area  to  read 
previous  messages 

•  Tap  OK  (in  upper-right  corner)  to  close 

•  Start  in  C2CE  Main  Window. 

•  Tap  Yellow  Pad  icon  (5th  from  left) 

•  You  can  display,  all  messages,  only  Transmitted 
(TX),  or  only  Received  (RX). 

•  List  shows  time  sent  or  received. 

•  Select  message  and  tap  View. 

•  Tap  Tx  to  transmit  selected  message  again. 

•  Can  scroll  and  adjust  column  widths. 

199 


Mapping 

|  Routes 

-  Start  in  C2CE  main  screen 
•  Icons  are  {left  to  right): 

—  Pointer  display  grid  of  the  location  you  tap 
—  Range  Bearing:  fromfto  two  points  on  map 
-Draw  (ignore) 

—  Route:  create  waypoints  in  a  route 
-  Pan:  tap  and  drag  to  move  tine  map  around 
—  Zoom  In  (+)3  Out(-),  Zoom  Box 

■  Tap  Route  icon  (R). 

■  Tap  points  along  route. 

*  Tap  Pointer  (arrow)  v/hen  done. 

*  Tap  Fits  menu  [at  bottom  of  screen),  select  Rts  Mgr 

-  Select  the  Q  ui  c  kRoute  you  ju  st  created  (highe  st  nu m  ber) 

*  Tap  Edit 

*  Use  Scroll  bar  and/or  adjust  column  width  to  see  the 
position  of  each  waypoint  on  the  route 

■  '^Course"  column  shows  the  bearing  from  this  waypoint  to 
the  next,  "Leg  Length1'  shows  the  distance. 

*  Total"  column  shoves  the  total  distance  up  to  that 
waypoint. 

Practical  Application  - 
Messaging 

Practical  Application  - 
Mapoina 

•  Start  C2CE,  close  C2GE,  start  it  again 

•  Send  a  comm  check. 

•  Send  three  and  receive  three  text 
messages. 

•  Recall  the  first  message  you  sent  (using 
the  message  log),  and  record  the  time  it 
was  sent.  Send  it  again. 

•  Find  the  grid  coordinate  of  a  specific  point 

•  Use  zoom  and  pan  to  find  various  points. 

•  Find  the  bearing  and  range  from  one  point 
to  another. 

•  Create  a  route  with  at  least  three  legs. 

•  Identify  the  bearing  and  distance  of  the 
first  leg. 

•  Find  the  total  distance  of  your  route. 

200 


5.  RPDA  AND  BASIC  LAND  NAVIGATION  PRACTICAL  APPLICATION 

Perform  each  task  using  both  conventional  means  (i.e.  paper  map  and  protractor) 
and  using  the  RPDA.  Write  down  notes  on  how  to  do  each  task,  and  bring  this 
sheet  with  you  to  the  testing. 

□  Start  RPDA  main  application,  close  it,  start  it  again. 

□  Send  communications  check 
With  a  partner: 

□  Send  three  and  receive  three  text  messages. 

□  Recall  the  first  message  you  sent,  and  record  the  time  it  was  sent: 


□  Send  the  first  message  you  sent  again. 

□  Find  the  grid  coordinate  of  a  specific  point,  use  zoom  and  pan  to  find  various 

points  (choose  two): _ _ 


Carmel  River  State  Beach 

Asilomar  State  Beach 

Monterey  Airport 

Salinas  Airport 

Moss  Landing 

□  Find  the  bearing  and  range  from  the  two  points  you  found  above. 
Bearing: _  Distance: _ 


□  Create  a  route  with  at  least  three  legs. 

□  Identify  the  bearing  and  distance  of  the  first  leg. 

Bearing: _  Distance: _ 

□  Find  the  total  distance  of  your  route. 

Total  Distance: 
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APPENDIX  H.  INFORMATION  FOR  PARTICIPANTS 


1.  RESEARCH  BRIEF 


© 

AWE 

Applied  Warfighter  Ergonomics 


Human  Factors 
Study  of 
Ruggedized 
Personal  Digital 
Assistants 
(RPDAs) 


© 

AWE 


Introduction 


•  What  this  study  is  (and  is  not) 

•  What  we  will  do  today 

•  What  we  will  do  later  this  week 

•  Admin  details 


Capt  Takagi,  USMC 
Lt  Read,  USAF 
Lt  Simpson,  USAF 


o 

AWE _ 


What  is  this  study  for? 


•  Testing  four  Ruggedized  Personal  Digital 
Assistants  (RPDAs)  in  use  or  being  considered 
by  US  Army  Special  Operations  Command 
(USASOC)  and  USMC  units  in  Iraq  and 
Afghanistan. 

•  We  are  primarily  testing  the  “user-friendliness”  of 
the  software  (not  the  hardware,  networking,  etc.) 

•  We  are  not  the  developers,  and  are  not 
advocates  of  any  particular  system. 

•  There  are  significant  limits  to  what  we  are 
studying,  but  we  will  report  all  comments  on  to 
the  program  managers. 


O 
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What  is  this  study  for? 


•  Why  so  few  devices? 

-  Most  are  committed  to  operational  forces. 

•  Why  does _ seem  broken? 

-  These  are  prototype  units. 

•  Will  I  ever  use  one  of  these? 

-  It’s  quite  possible.  For  example,  there  are 
variations  specialized  for  translation. 
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haB  Today 

Later  this  week... 

•  2-3  hour  training  session  on  a  single 

•  Each  participant  will  come  to  NPS  to 

Ruggedized  Personal  Digital  Assistant 

operate  the  RPDA  for  a  single  1-2  hour 

(RPDA). 

individual  session,  which  will  be 

•  Review  of  similar  tasks  in  the  “traditional” 

videotaped. 

method  (radios,  paper  maps). 

•  Will  schedule  small  groups  at  one  time 

•  Schedule  testing  sessions  for  later  this 

(prefer  2-6  individuals). 

week. 

•  Lab  is  available  0700-0900  and  1000-1800 

Wed  and  Thu,  and  0700-1900  Friday. 

0 
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Admin 


O 
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Privacy  and  Safety 


•  Any  questions  about  your  packet? 

-  Keep  these: 

•  Participant  Information  Form 

•  Card  with  your  Participant  ID# 

-  Turn  in  when  you  come  to  NPS  (not  today) 

•  Three  consent  forms  (signed) 

•  Participant  ID  form  ("For  Official  Use  Only") 

•  Technology  Experience  Survey 


•  Your  name  will  not  be  used  anywhere  in  the 
study  or  the  final  report. 

•  Please  speak  freely  so  that  your  feedback  can 
be  used  to  improve  the  devices. 

•  If  you  choose  not  to  participate,  simply  let  your 
NCOs  know.  I  will  not  know  who  did  or  did  not 
participate  in  the  study. 

•  During  this  study,  there  should  be  no  more  risk 
to  you  than  you  are  exposed  to  in  your  current 
daily  routine.  Please  inform  any  researcher  if 
you  see  any  unsafe  conditions. 
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2.  LETTER  TO  REQUEST  PARTICIPANTS 


Applied  Warfighter  Ergonomics  /  Human  Systems  Integration 

Department  of  Operations  Research  Naval  Postgraduate  School 

Glasgow  Hall  rm  221 

lab  phone:  (831)656-3134 


I  am  a  graduate  student  conducting  Human  Factors  research  at  the  Naval  Postgraduate  School. 
We  are  seeking  participants  in  equipment  testing.  Specifically,  we  are  testing  ruggedized 
personal  digital  assistants  (RPDAs  -  basically  handheld  networked  tactical  computers)  for  the 
Special  Operations  Command  (SOCOM).  Some  of  the  devices  are  in  current  use  by  SOCOM  in 
Iraq  and  Afghanistan. 

We  are  seeking  DLI  students,  available  between  now  and  mid-August,  to  participate  in  this 
important,  real-world  testing.  They  would  definitely  benefit  from  exposure  to  state-of-the-art 
equipment  that  is  already  filtering  down  from  SOCOM  to  regular  units,  and  by  participating  in  the 
tactical  scenarios  (based  on  current  operations  in  Iraq  and  elsewhere)  that  we  are  developing. 

Our  testing  would  involve: 

1 .  Approximately  two  hours  of  training  on  the  equipment  in  a  group  session. 

2.  Approximately  one  hour  per  individual  to  observe  the  students,  one  at  a  time,  using  the 
equipment  in  simulated  tactical  scenarios. 

Ideally,  participants  would  be  military  personnel  with  very  basic  training  in  tactical  procedures, 
such  as  wearing  body  armor,  reading  a  map,  and  talking  on  a  radio.  The  training  any  Marine  or 
Soldier  receives  in  boot  camp  would  be  sufficient.  Personnel  from  other  services,  and  even 
civilians  (including  those  who  have  never  been  exposed  to  any  tactical  training)  would  also  be 
welcome.  Participants  should  not  be  "specially  selected"  on  the  basis  of  being  more  or  less 
proficient  than  others  -  the  equipment,  not  the  participants,  is  being  tested.  The  participants 
should  be  representative  of  all  possible  users,  skilled  and  less  skilled. 

To  be  a  rigorous  test,  we  would  like  to  have  twenty  to  forty  individuals,  who  would  be  available  as 
described  above.  Each  individual  participant  would  only  have  to  be  available  for  the  initial  half-day 
training  in  small  groups,  and  then  for  a  few  hours  sometime  in  the  next  week  or  so.  Since  each 
participant  must  be  observed  and  recorded  individually,  it  will  take  several  days  to  work  through 
all  the  participants,  one  at  a  time. 

We  are  very  flexible  about  when  and  how  this  could  occur,  between  now  and  maybe  mid-August. 
We  could  also  do  this  as  a  series  of  smaller  groups  (as  few  as  four  to  ten  individuals,  in  several 
blocks  over  the  period),  if  that  would  be  easier  to  schedule. 

The  attachment  describes  our  laboratory  equipment.  I  would  be  happy  to  demonstrate  our 
laboratory,  or  brief  this  experiment,  to  anyone  who  may  be  interested  in  more  information.  I  can 
best  be  contacted  via  email:  ktakaqi@nps.edu. 

Very  Respectfully, 

Koichi  Takagi 
Capt  USMC 

email:  ktakaqi@nps.edu 
home:  (831)372-5912 
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3. 


PARTICIPANT  INFORMATION  FORM 


To  all  research  study  participants: 

You  are  invited  to  participate  in  a  research  study  at  the  Naval  Postgraduate  School.  Your 
participation  will  help  improve  equipment  currently  in  use  by  Special  Forces  in  Iraq  and 
Afghanistan.  Please  read  the  description  of  the  research  on  the  following  three  pages,  and  initial 
and  sign  where  requested. 

This  is  not  a  test  of  your  skills  or  abilities.  I  want  to  emphasize  that  your  participation  or 
non-participation  in  this  study  will  have  no  effect  on  your  current  or  future  military  status.  The 
equipment,  not  the  individual  participants,  is  being  evaluated.  There  is  no  question  of  you,  the 
participant,  doing  "well,"  or  "poorly,"  and  neither  your  participation  nor  your  individual  results  will 
be  entered  into  your  military  record,  or  released  to  your  current  command. 

Please  do  not  discuss  any  part  of  the  study  with  other  students,  or  potential  participants, 
until  the  study  has  concluded  (around  September,  2005).  We  want  all  participants  to  be  equally 
prepared  during  the  study,  and  no-one  to  be  "coached"  or  otherwise  prepped  beforehand. 

During  the  study,  you  will  receive  approximately  two  to  four  hours  of  training  (as  a  group) 
on  a  piece  of  military  equipment.  Please  do  your  best  to  learn  the  gear,  and  ask  as  many 
questions  as  you  like  of  the  instructors  or  researchers.  Within  a  few  days  after  the  training,  you 
will  be  brought  to  the  Naval  Postgraduate  School  for  an  individual  session  of  less  than  two  hours. 
You  will  operate  the  equipment  you  were  trained  on  while  wearing  body  armor,  helmet,  etc.,  and 
watching  a  tactical  scenario  on  a  large  plasma-screen  TV.  Many  portions  of  the  study  will  be 
videotaped,  and  so  there  is  a  release  form  that  we  ask  you  to  sign  so  that  we  may  use  the 
recordings.  These  standard  consent  forms  are  always  required  in  any  kind  of  research  on  people. 
I  do  not  believe  there  is  the  slightest  risk  to  your  health  or  to  your  privacy. 

Please  fill  out  the  enclosed  survey.  If  you  have  any  questions  about  the  survey,  we  will 
answer  them  during  the  group  training  session. 

Please  don't  hesitate  to  ask  us  any  questions  you  may  have  at  any  time  during  the  study. 
Thank  you  for  your  participation. 

Sincerely, 


Capt  Koichi  Takagi 
United  States  Marine  Corps 
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4. 


CONSENT  FORMS 


PRIVACY  ACT  ST  ATM  ENT 

NA  VAL  POSTGRADUATE  SCHOOL,  MONTEREY,  CA  93943 
PRIVA  CY  ACT  STA  TEMENT 

1  .  Purpose:  Data  from  this  research.  Evaluating  the  Human  Factors  of  Military  Handheld 
Computing  Devices,  will  be  used  to  develop  tests,  tactics,  procedures,  and  equipment  to 
improve  the  human-machine  interfaces  of  military  handheld  computing  devices. 

2,  Use:  Research  data  from  this  study  will  be  used  for  statistical  analysis  by  the  Departments 
of  the  Navy  and  Defense,  and  other  U.S.  Government  agencies,  provided  this  use  is 
compatible  with  the  purpose  for  which  the  information  was  collected.  Use  of  the 
information  may  be  granted  to  legitimate  non-go vemmenl  agencies  or  individuals  by  the 
Naval  Postgraduate  School  in  accordance  with  the  provisions  of  the  Freedom  of  Information 
AcL 

3.  Disclosure/Confidentiality: 

a.  I  have  been  assured  that  my  privacy  will  be  safeguarded.  I  will  be  assigned  a  control  or 
code  number  which  thereafter  will  be  the  only  identifying  entry  on  any  of  the  research 
records.  The  Principal  Investigator  will  maintain  the  cross-reference  between  name  and 
control  number.  It  will  be  decoded  only  when  beneficial  to  me  or  if  some 
circumstances,  which  is  not  apparent  at  this  time,  would  make  it  clear  that  decoding 
would  enhance  the  value  of  the  research  data.  In  all  cases,  the  provisions  of  the  Privacy 
Act  Statement  will  be  honored. 

b«  I  understand  that  a  record  of  the  infonnation  contained  in  this  Consent  Statement  or 
derived  from  the  experiment  described  herein  will  be  retained  permanently  at  the  Naval 
Postgraduate  School  or  by  higher  authority.  I  voluntarily  agree  to  its  disclosure  to 
agencies  or  individuals  indicated  in  paragraph  3  and  I  have  been  informed  that  failure  to 
agree  to  such  disclosure  may  negate  the  purpose  for  which  the  experiment  was 
conducted. 

c.  I  also  understand  that  disclosure  of  the  requested  information,  including  my  Social 
Security  Number,  is  voluntary , 


Name,  Ranh  (please  print) 

Signature  of  Voluntary  Participant  Date 
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PARTICIPANT  CONSENT  FORM 


1  Inmgi:  Introduction.  You  are  invited  to  panic  spate  in  a  study  of  military  handheld  computing  devices.  With 

information  gathered  from  you  and  other  participants,  we  hope  to  discover  the  human  factors  considerations 
pertaining  to  these  devices.  Please  read  and  sign  this  form  indicating  that  you  agree  to  be  in  the  study.  Feel  free  to 
ask  any  questions  you  may  have  before  signing. 

2.  initial: _ Background  Information.  The  Naval  Postgraduate  School  Human  System  Integration  Laboratory 

and  Applied  Warfighter  Ergonomics  Center  are  conducting  this  study. 

3.  initiate _ Procedures.  If  you  agree  to  participate  in  this  study,  the  researcher  will  explain  the  tasks  in  detail. 

There  will  be  several  sessions  consisting  of  written  questionnaires,  equipment  training  and  operation,  and 
participation  in  multimedia  tactical  scenarios, 

4.  initial: _ Risks  and  Benefits,  This  research  involves  no  risks  or  discomforts  other  than:  answering  questions 

(on  paper  and  in  person)  about  your  experiences  w  ith  various  equipment  during  field  exercises  and/or  laboratory 
trials.  You  may  be  asked  to  wear  standard  military  individual  equipment.  Audio  and  video  recordings  or  your 
participation  will  be  made  if  you  consent  (item  9  below).  Your  participation  will  significantly  contribute  to 
improving  the  effectiveness  of  equipment  currently  being  fielded  in  the  US.  Dept,  of  Defense  and  to  current 
research  in  human -machine  interaction. 

5.  Initial:  Compensation.  No  tangible  reward  will  be  given.  A  copy  of  the  results  wifi  be  available  to  you  at  the 

conclusion  of  the  experiment. 

6.  initial: _ Confidentiality.  The  records  of  this  study  will  be  kept  confidential.  Special  care  will  be  taken  to 

preserve  anonymity  in  all  data  collection.  Written  documentation  will  be  identified  by  participant  number  only. 
No  information  which  could  identify  you  as  a  participant  will  be  made  accessible  to  your  military  unit,  your 
supervisors,  or  to  the  public. 

7.  Initial: _ Voluntary  Nature  of  the  Study.  If  you  agree  to  participate,  you  are  free  to  withdraw  from  ihq  study 

at  any  time  without  prejudice.  You  will  be  provided  a  copy  of  this  form  for  your  records. 

S,  Initial:, _ _ Points  of  Contact.  If  you  have  any  further  questions  or  comments  after  the  completion  of  the  study, 

you  may  contact  the  principal  researcher,  Nita  Lewis  Miller,  PhD,  at  (E31)  656-2281,  DSN:  756-2281, 
nlmi  11  ertrfnps.edu,  or  Capt  Koichi  Takagi,  USMC,  at  (831)  656-3134,  ktakatiifa/ nrs.edu. 

9.  Initial: _ Consent  to  Video  and  Audio  Recording.  T  agree  that  audio  and  video  recordings  of  my  participation 

in  this  research  may  be  made.  1  agree  that  the  recordings  may  be: 

initial:  Studied  by  members  of  the  research  team, 

initial:. _ _  Shown  to  other  researchers,  research  participants,  and  students,  and 

Initial: _ Used  in  future  training. 

10.  initial: _ Statement  of  Consent,  I  have  read  the  above  information.  Any  questions  I  had  concerning  this 

research  have  been  answered  to  my  satisfaction.  I  agree  to  participate  in  this  study. 


Signature  erf  Researcher 


Signature  of  Voluntary  Participant 


I^TjJua 
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MINIMAL  RISK  CONSENT  STATEMENT 

NAVAL  POSTGRADUATE  SCHOOL,  MONTEREY,  CA  93943 

Participant:  VOLUNTARY  CONSENT  TO  BE  A  RESEARCH  PARTICIPANT  IN:  Evaluating  the  Human  Factors 
of  Military  Handheld  Computing  Devices 


1 .  I  have  read,  understand  and  been  provided  "Information  for  Participants"  and  "Participant  Consent 
Form"  that  provides  the  details  of  the  below  acknowledgments. 

2.  I  understand  that  this  project  involves  research.  An  explanation  of  the  purposes  of  the  research,  a 
description  of  procedures  to  be  used,  identification  of  experimental  procedures,  and  the  extended 
duration  of  my  participation  have  been  provided  to  me, 

3.  1  understand  that  this  project  involves  only  minimal  risk,  1  have  been  informed  of  any  reasonably 
foreseeable  risks  or  discomforts  to  me. 

4  I  have  been  informed  of  any  benefits  to  me  or  to  others  that  may  reasonably  be  expected  from  the 
research. 

5.  I  have  signed  a  statement  describing  the  extent  to  which  confidentiality  of  records  identifying  me  will 
be  maintained. 

6.  1  have  been  informed  of  any  compensation  and/or  medical  treatments  available  if  injury  occurs  and  is 
so,  what  they  consist  of,  or  where  further  information  may  be  obtained. 

7 .  T  understand  that  my  participation  in  this  project  is  voluntary,  refusal  to  participate  will  involve  no 
penalty  or  loss  of  benefits  to  which  I  am  otherwise  entitled.  I  also  understand  that  I  may  discontinue 
participation  at  any  time  without  penally  or  loss  of  benefits  to  which  I  am  otherwise  entitled. 

8.  1  understand  that  the  individual  to  contact  should  I  need  answers  to  pertinent  questions  about  the 
research  is  Professor  Nita  L.  Miller,  Principal  Investigator,  and  about  my  rights  as  a  research 
participant  or  concerning  a  research  related  injury  is  Prof.  Jim  Eagle,  Operations  Research  Dept. 
Chairman  or  LTC  Erie  Morgan,  MC,  USA,  Commanding  Officer  of  the  Presidio  of  Monterey  Health 
Clinic.  A  full  and  responsive  discussion  of  the  elements  of  this  project  and  my  consent  has  taken 
place.  NFS  Medical  Monitor:  LTC  Eric  Morgan,  MC,  USA,  er i c .morgan ftfcn w , amed d . arm v.m i  1 . 
(831)  242-7550,  doctor  on-call  pager:  (831)  648-2177. 


Signature  of  Researcher 

Date 

'g-jgw/az&m wmm 

■■■KSES^I 

Signature  of  Voluntary  Participant 

Date  I 
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5. 


MAPS  TO  NPS 


Enter  the  10th  St  and  Sloat 
Follow  perimeter  road:  Morse  Rd.  turns  into 
Lake  Del  Monte,  turns  into  Cunningham  Rd. 
Turn  Right  on  Bouldry  Road  and  look  for 
parking.  Parking  is  very  difficult  on  NPS  - 
allow  15  min.  just  to  find  a  space! 

Lab  is  in  Glasgow  Hall  rm  221,  ph  656-3134. 
Capt  Takagi  cell  phone  236-6306. 
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6. 


PARTICIPANT  ID  FORM  AND  ID  CARDS 


FOR  OFFICIAL  USE  ONLY 

This  document  contains  personally  identifying  research  information  and 
may  only  be  viewed  by  authorized  personnel  associated  with  this  research. 


Participant  ID  # 

Date 

Please  keep  your  participant  ID  #  with  you  at  all  times  until  the  conclusion 
of  the  study. 

The  information  below  will  be  used  only  if  there  is  a  need  to  contact  you 
after  the  conclusion  of  this  study.  Special  care  will  be  taken  to  preserve 
anonymity  during  all  data  collection  and  analysis.  Any  comments  or  opinions  you 
express  during  this  study  will  not  be  attributed  to  you  in  any  manner.  Written 
documentation  will  be  identified  by  your  Participant  ID  number  only,  and  faces 
and  voices  will  be  obscured  on  video  recordings  when  possible.  No  information 
which  could  identify  you  as  a  participant  will  be  made  accessible  to  your  military 
unit,  your  supervisors,  or  to  the  public  without  your  consent. 


Last  name 

First  name,  middle 
initial 

Rank 

Branch  of  service 

Unit  or 
organization 
POC  information 

please  give  a  mailing 
address,  and  phone  number  that 
we  could  use  to  contact  you  in 
the  future,  if  necessary 

email  address 

FOR  OFFICIAL  USE  ONLY 

This  document  contains  personally  identifying  research  information  and 
may  only  be  viewed  by  authorized  personnel  associated  with  this  research. 
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Research  Participant 
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Applied  Warfighter  Ergonomics 
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APPENDIX  I.  INSTITUTIONAL  REVIEW  BOARD  FORMS 


Nka  Lewis  Miller,  Ph.D* 
Operations  Research  Department 
Glasgow  Hall 

Naval  Postgraduate  School 
Monterey,  California  93943 


531-656-2281 
DSN:  756-2281 
Fax:  831-656-2595 
nlmiller@npsHnaiT.mil 


To :  P  rot  ect  ion  of  Hum  an  Subj  ect  s  Committee 

Subject:  Application  for  Human  Subj  ects  Review  for  Evaluating  the  Human  Factors  of  Military 

Handheld  Computing  Devices 

1 .  Attached  is  a  set  of  documents  outlining  surveys,  direct  observation  and  recording, 
interviews,  after-action  reports,  and  focus  group  studies  to  be  conducted  over  the  next  year. 
Some  studies  will  be  conducted  in  conjunction  with  the  Tactical  Network  Topology  (TNT) 
project  at  NPS,  Inter-4  Corporation  of  San  Francisco,  California,  Exponent  Corporation  of 
Menlo  Park,  California,  the  United  States  Army  Special  Operations  Command  (USASOC), 
and  the  United  States  Marine  Corps, 

2.  We  are  requesting  approval  of  the  described  experimental  protocol.  An  experimental  outline 
is  included  for  your  reference  that  describes  the  methods  and  measures  we  plan  to  use. 

3.  We  include  the  consent  forms,  privacy  act  statements,  all  materials  and  forms  that  a  subject 
will  read  or  fill-out,  and  the  debriefing  forms  (if  applicable)  we  will  be  using  in  the 
experiment. 

4.  We  understand  that  any  modifications  to  the  protocol  or  instruments/measures  will  require 
submission  of  updated  rRB  paperwork  and  possible  re-review.  Similarly,  we  understand  that 
any  untoward  event  or  injury  that  involves  a  research  participant  will  be  reported 
immediately  to  the  TRB  Chair  and  NFS  Dean  of  Research. 


Nita  Lewis  Miller,  PhD 


Koichi  Takagi,  Capt  USMC 
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APPLICATION  FOR 
HUMAN  SUBJECTS  REVIEW  (HSR) 


HSR  NUMBER 


(to  be  assigned) 


PRINCIPAL  INVESTIGATOR(S)  (Full  Name,  Code,  Telephone) 

Nita  Lewis  Miller,  Ph.D.,  CODE  61  831-656-2281  DSN:  756-2281  Fax:  831-656-2595 
Koichi  Takagi,  Capt,  USMC,  CODE  61  831-656-2281  DSN:  756-2281  Fax:  831-656-2595 


APPROVAL  REQUESTED  [  X  ]  New  [  ]  Renewal 


LEVEL  OF  RISK  (  )  Exempt  [  X  ]  Minimal  [  ]  More  than  Minimal 
Justification: 


WORK  WILL  BE  DONE  IN  (Site/Bldg/Rm) 

Glasgow-  221,  Camp  Roberts,  California,  Fort 

Ord,  California,  and  other  field  locations 

ESTIMATED  NUMBER  OF  DAYS  TO  COMPLETE 

1 80  days 

MAXIMUM  NUMBER  OF  SUBJECTS 

100 

ESTIMATED  LENGTH  OF  EACH  SUBJECT’S 
PARTICIPATION 

est.  total  6  hrs,  2-4  hrs  per  day,  in  a  2  week  period 

SPECIAL  POPULATIONS  THAT  WILL  BE  USED  AS  SUBJECTS 

(  ]  Subordinates  [  ]  Minors  [  X  ]  NPS  Students  [  ]  Special  Needs  (e.g.  Pregnant  women) 

Specify  safeguards  to  avoid  undue  influence  and  protect  subject’s  rights: 

Special  care  will  be  taken  to  preserve  anonymity  in  all  data  collection.  Written  documentation  will  be 
identified  by  number  only,  and  faces  and  voices  may  be  obscured  on  video  recordings. _ 

OUTSIDE  COOPERATING  INVESTIGATORS  AND  AGENCIES 

Tactical  Network  Topology  (TNT)  project  at  NPS,  Inter-4  Corporation  of  San  Francisco,  California, 
Exponent  Corporation  of  Menlo  Park,  California,  the  United  States  Army  Special  Operations 
Command  (USASOC),  and  the  United  States  Marine  Corps. _ 

TITLE  OF  EXPERIMENT  AND  DESCRIPTION  OF  RESEARCH  (attach  additional  sheet  if  needed). 
.Army  Rangers,  Special  Forces  team  members,  and  other  DoD  enlisted  members  and  officers,  will  be 
field  testing  new  handheld  computers  (including  lnter-4's  Tacticomps,  NavBoards  and  Exponent  .ARC 
PDAs)  and  the  Tactical  Network  Topology  (TNT)  system.  Other  handheld  systems  may  be  tested  in 
similar  environments,  as  available.  A  short  pre-excrcise  will  be  given  to  assess  each  participant's 
familiarity  and  comfort  in  using  current  technology.  During-  and  post-exercise  surveys,  interviews,  and 
focus  groups  will  assess  the  performance  and  usability  of  the  equipment.  Participants'  interactions  with 
the  equipment  may  be  recorded  (audio  and  video)  during  field  exercises  and  in  laboratory  settings,  and 
detailed  records  (behavioral  coding)  made  of  their  activities  and  reactions.  The  usability  of  military 
handheld  electronic  devices,  and  the  factors  influencing  the  acceptance  and  adoption  of  those  devices, 

will  be  analyzed  as  a  product  of  this  research. _ 

I  have  read  and  understand  NPS  Notice  on  the  Protection  of  Human  Subjects.  If  there  are  any  changes 
in  any  of  the  above  information  or  any  changes  to  the  attached  Protocol,  Consent  Form,  or  Debriefing 
Statement,  I  wilfsu^end  the  experimenumtil  I  obtain  new  Committee  approval. 

SIGNATURE  _  DATE 

SIGNATURE  ^  V ^  T*- _  DATE 
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APPENDIX  J.  SOFTWARE  AND  HARDWARE  CONFIGURATION 


1 .  NOLDUS  "THE  OBSERVER"  SOFTWARE 

Observation  files  were  named  using  this  convention: 

R _ 

The  first  letter  after  "R"  is  the  device  tested,  either  A-D  for  the  RPDAs,  or  Z 
for  the  conventional  equipment. 

The  next  four  digits  are  the  participant's  ID,  in  the  range  0500-0599  for  this 

study. 

The  last  two  character's  are  the  coder's  initials,  either  "KT"  for  Koichi 
Takagi  or  "KB"  for  Katie  Buettner. 
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X-KEYS  MACRO  KEYBOARD 


In  addition  to  the  key  assignments  shown  below,  the  jog  dial  (inner 
spinning  wheel)  was  set  to  FFWD  (jog  right)  and  REW  (jog  left). 


X-keys  Macro  Maker  -  Takagi  RPDA  study  conf.xk4  Editor,  Green  layer 


Key  Type  Comment 

Key  Type  Comment 

Key  Type  Comment 

0  Macro  edit  obs. 

30  Macro  OFF  2c  plot  rte 

60  Macro 

1  Macro  «  REW 

31  Macro  tactical 

61  Macro  -4x 

2  Macro  FFWD  » 

32  Macro  admin 

62  Macro  -2x 

3  Empty 

33  Macro  comment 

63  Macro  -lx 

4  Macro  PLAY  > 

34  Empty 

64  Macro  -1/2x 

5  Macro  PAUSE 

35  Empty 

65  Macro  -1/5x 

6  Macro  STOP 

36  Macro  ON  2a  plot  rte 

66  Macro  -1/25x 

7  Macro  Quick  Rev 

37  Macro  OFF  2a  plot  rte 

67  Empty 

8  Empty 

38  Macro  ON  3c  read  grid 

68  Macro  1/25x 

9  Empty 

39  Macro  OFF  3c  read  grid 

69  Macro  1/5 

10  Empty 

40  Macro  ON  4c  brg  rng 

70  Macro  1/2 

11  Empty 

41  Macro  OFF  4c  brg  rng 

71  Macro  lx 

12  Empty 

42  Macro  ON  5c  send  msg 

72  Macro  2x 

13  Empty 

43  Macro  OFF  5c  send  msg 

73  Macro  4x 

14  Empty 

44  Macro  ON  6c  rev  msg 

74  Macro  FFWD 

15  Empty 

45  Macro  OFF  6c  rev  msg 

75  Macro  Play 

16  Empty 

46  Macro  ON  7c  read  back 

76  Macro 

17  Empty 

47  Macro  OFF  7c  read  back 

77  Macro 

18  Empty 

48  Macro  ON  3a  read  grid 

78  Macro 

19  Empty 

49  Macro  OFF  3a  read  grid 

79  Macro 

20  Macro  ON  1c  comm  chk 

50  Macro  ON  4a  brg  rng 

80  Macro 

21  Macro  OFF  1c  comm  chk 

51  Macro  OFF  4a  brg  rng 

82  Empty 

22  Macro  slip 

52  Macro  ON  5a  send  msg 

83  Macro  Still 

23  Macro  mistake 

53  Macro  OFF  5a  send  msg 

84  Macro 

24  Macro  device  error 

54  Macro  ON  6a  rev  msg 

85  Macro 

25  Macro  interface  error 

55  Macro  OFF  6a  rev  msg 

86  Macro 

26  Macro  equipment  handling  error 

56  Macro  ON  7a  read  back 

87  Macro 

27  Macro  ON  la  comm  chk 

57  Macro  OFF  7a  read  back 

88  Macro 

28  Macro  OFF  la  comm  chk 

58  Macro  FFWD 

89  Macro  Play 

29  Macro  ON  2c  plot  rte 

59  Macro  RWD 
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APPENDIX  K.  DATA 


1.  FOCUS  GROUP  NOTES 

RPDA  Focus  Group 

Ad-hoc,  non-attributional  military  RPDA  Focus  Group  conducted  1600  -  1730  on 
22  Feb  2005  at  Camp  Roberts,  CA.  Facilitator:  Captain  Koichi  Takagi,  USMC, 
student  researcher  at  the  Naval  Postgraduate  School 


Participants: 

Three  senior  Special  Forces  staff  NCOs  participating  in  a  military  PDA  evaluation 
exercise. 

One  representative  from  USASOC  program  office  (but  not  the  Mobile  Access 
Infrastructure  program  office  responsible  for  military  PDAs). 

One  employee  of  an  RPDA  development  company. 


Purpose:  to  determine  possible  requirements  and  capabilities  for  handheld 
military  computing  devices  (military  PDAs).  A  “brainstorming”  environment  during 
the  focus  group  was  sought.  Ideas  were  encouraged  regardless  of  current 
feasibility  or  real  applicability.  The  following  is  a  list  of  topics  discussed,  including: 
possible  operational  requirements  for  military  PDAs,  general  and  specific 
functionality  that  military  PDAs  might  have,  possible  uses  for  and  tactical 
employment  of  military  PDAs,  and  random  comments.  Some  participants  had  not 
used  military  or  civilian  PDAs  to  any  great  extent,  some  were  unfamiliar  with  the 
military  tasks  discussed.  Non-attribution  was  emphasized,  and  it  was  understood 
that  the  discussion  would  not  generate  obligations  by  the  contractor  or  the 
government. 
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Four  critical  requirements  defined  as  broad  functionality  required  of  a  military 
PDA: 

Text  messaging,  Voice  over  IP  (VOIP)  capability,  Moving  video  and  still  imagery 
dissemination,  and  GPS  -  mapping  -  situational  awareness  (SA)  -  “Blue  Force 
Tracker  (BFT)”  capability. 

Text  messaging 

ease  of  input  (keyboard,  on-screen  keyboard,  stylus,  or  other) 

speed  and  accuracy  of  input  are  crucial  -  must  test  error  rate  and  speed 

input  must  be  reliable  under  a  wide  range  of  operating  environments  (bumpy 
vehicle,  day,  night,  one  handed,  etc.) 

voice  recognition?  Again,  must  be  reliable  under  combat  conditions  -  injury, 
illness,  extreme  background  noise 

cut  &  paste  between  messages  sent  and/or  received 

word  wrap  so  that  whole  message  is  always  visible,  rather  than  a  scrolling  single 
line 

ability  to  change  text  size  for  different  conditions  (bumpy  driving,  day,  night,  etc.) 

lexicon  of  standard  military  terms  and  abbreviations 

auto-correct  and  spell  check  based  on  above  lexicon 

auto-complete  -  this  might  be  bad  if  it  made  dangerous  errors 

recent  word  history 

cues  for  message  received  -  audio,  visual,  vibrate?  Be  able  to  turn  off  alerts 
message  sent  confirmation  (is  network  up?  Did  I  do  it  right?) 
delivery  confirmation  (did  they  get  it?) 
keep  chat  open  but  minimized 

advantage  -  text  chat  is  1000  times  less  bandwidth  than  voice. 

Things  we  would  or  could  do  with  text  messaging: 

•  Lists  of  info  to  save  for  later  -  convoy  routes,  admin  details,  fragOs,  op 
order 

•  Distribution  groups  for  an  entire  unit 

•  Add  annotation  to  pictures,  describe  details 

•  Flag  messages  as  important  for  later  review  /  action 
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Retrieve  past  messages,  edit  and  send  on  to  others 


How  would  we  do  this  without  a  text-messaging  military  PDA 
Other  data  means  -  SMEARS?  Mini  laptop/palmtop  VDC  card  ViaSat 


Voice  over  IP  (VOIP) 

Important:  quality  of  transmission,  reliability,  clarity  (comparable  to  current 
systems) 

Full  duplex?  Don’t  see  need 
Record/playback  capability 

Probably  won’t  replace  any  current  voice  radio  systems  (MBTRs,  SINCGARS, 
etc.)  for  a  long  time,  so  would  be  an  additional  piece  of  gear  to  carry  just  for  voice 
comm.  Because  of  this,  is  VOIP  really  a  pressing  need?  We  will  still  use  many 
other  ways  to  talk. 

Used  for: 

•  Emergency  /  backup  long-range  comm,  reach-back  to  higher  command 

•  Flexibility,  alternate  means 

•  Collaboration  while  viewing  images  (Groove) 

•  Mobile  whiteboard  -  annotate  images,  drawings,  while  talking  about  it 
Video  and  Still  imagery  dissemination 

Requirements:  minimum  5  fps,  minimize  jitter  and  data  loss 

Network  needs  to  be  able  to  provide  a  minimum  of  3  simultaneous  video  streams 
(video  sources) 

Quality  is  affected  by  sensor  capabilities  (color,  contrast,  zoom) 

Current  streaming  video  is  “quarter  VGA”  (QVGA)  =  320x240  pixels  (similar  to 
VHS  tape  or  broadcast  TV) 

Would  like  a  local  buffer  /  save  capability  beyond  just  streaming  so  we  can 
playback,  rewind,  etc. 

Search  &  Rescue  “panic  button”  to  show  unit  what’s  happening  if  I  go  down. 

File  type  compatibility  has  not  been  an  issue 

Different  client  users  involved:  dedicated  intel  analysts  (formally  school-trained  or 
not),  and  untrained  regular  operators.  These  different  users  have  different 
expectations  and  capabilities  when  given  the  same  image  -  should  test 
separately. 
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We  can  measure  things  like  frame  rate,  number  of  pixels,  network  jitter,  etc.,  but 
the  final  measure  of  success  is  subjective  -  can  the  user  correctly  interpret  the 
video  presented? 


Mapping,  GPS,  situational  awareness,  Blue  Force  Tracker  (BFT) 

System  should  replace  BFT,  SA  tools,  MTS 
Should  completely  replace  PLGRS 

BFT  /  SA  must  have  restricted  lists  for  SOF  teams  -  not  everyone  is  allowed  to 
see  where  they  are 

Mapping  -  how  does  interpreting  an  on-screen,  2inch  by  3inch  map  compare 
mentally  to  reading  a  paper  map? 

Paper  provides  far  higher  resolution  (2400dpi+  compared  to  72dpi  on  screen), 
larger  area,  quicker  and  more  intuitive  manipulation,  easier  stowage. 

Computer  capability  -  have  a  whole  map  library  in  your  pocket,  from  1:250,000 
down  to  room-by-room  blueprints. 

Should  provide  all  standard  map  overlay  information  like  routes,  NAIs,  RFAs,  etc. 
Filters  for  just  the  units  and  info  I  want  to  know. 

Secure  comm,  critical 

Need  to  have  some  positive  indicator  that  the  units  and  overlay  information 
shown  is  reliable  or  not,  is  old  or  new,  and  how  precise  it  is. 

Time  of  last  update  shows  as  a  pop-up? 

Icons  change  color  as  they  “age”? 

Should  map  also  show  when  you  are  off  the  network,  like  C2PC 

If  there  is  nothing  shown  on  my  map,  am  I  missing  info,  off  the  net,  or  is  the  area 
truly  clear? 

Indications  of  what  we  know,  what  we  don’t  know,  etc. 

Ultimate  measure  of  success:  user’s  ability  to  understand  the  info  presented  and 
translate  it  into  ground  truth. 

Everyone  looks  at  a  map  and  a  situation  differently.  Must  be  able  to  customize 
view  for  each  individual  operator. 

Common  iconology 

Classic  icons  used  on  paper  don’t  always  translate  well  to  lo-res  screen. 
Example  -  checkpoint  symbol  is  usually  very  hard  to  see  on  screen  because  it’s 
so  small.  Unit  symbols  are  hard  to  interpret  well.  Maybe  it’s  time  for  new  icons  for 
digital  maps. 
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2.  SURVEY  DATA:  INDIVIDUAL  COMMENTS 
RPDA  A  -  Comments  from  Handheld  Device  Evaluations 

6.  Satisfaction  With  Text  Messages  Sent 

(501 )  4-  easy  to  hit  wrong  key  while  typing 

(502)  4-  keypad  letters  small,  but  usable 

(506)  4-  voice  recognition  or  a  small  keypad  would  make  entries  easier  than  with 
the  stylus;  transmission  was  easy  to  effect 

8.  Satisfaction  With  Text  Messages  Received 

(502)  5-  would  prefer  notification  when  message  received 
(506)  5-  easy  to  read 

17.  Satisfaction  With  Mapping  Functions 

(501)  3-  finding  labeled  places  was  easy,  but  finding  other  locations  was  hard 

(502)  4-  easy  to  view  self  and  others;  some  symbols  hard  to  interpret  (road, 
airport) 

(506)  2-  small  screen  made  it  difficult  to  orient  self  quickly 

18.  Satisfaction  With  SA  Tools 

No  comments 

20.  Satisfaction  With  Stylus 

(501)  1-  hard  to  use  the  stylus  to  scroll  around  map 

(505)  5-  touch  keypad  small  and  possible  to  make  mistakes,  but  overall  good 
feature 

(506)  4-  stylus  ok,  but  would  prefer  keypad  or  number  pad  like  on  a  cell  phone 

26.  Ability  to  Use  Personal  Weapon 

(501)  1-  do  not  see  how  you  could  operate  Ml 6  and  RPDA 

(502)  1-  only  able  to  hold  weapon  with  one  hand 

(505)  2-  hard  to  hold  RPDA  and  maintain  good  grip  on  weapon 

(506)  2-  cumbersome;  RPDA's  wires  got  tangled  with  weapon  sling 

28.  Overall  Satisfaction  With  Reliability 

(501)  2-  reliable  for  sending  messages,  but  some  issues  with  stylus  navigation  of 
the  map 

(502)  5-  no  malfunctions;  confident  in  RPDA's  calculations 
(506)  5-  no  errors  or  malfunctions 

30.  Overall  Satisfaction  With  System 

(501)  2-  stylus  tricky  to  use  in  map  mode  and  RPDA  gear  bulky  and  harder  to 
move  with 
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(502)  5-  helpful  in  eliminating  human  error  in  calculating  brg/rng;  text  messaging 

quieter  than  radio  for  covert  situations 

(506)  4-  effective  communication,  but  hard  to  use  quickly 

31.  Impression  of  Ease  of  Learning  and  Use 

(502)  5-  extremely  easy 
(506)  5-  easy  to  operate 

32.  Free  Comments 

(501)  separate  backpack  and  wires  make  RPDA  A  less  desirable 

(502)  helpful  if  equipment  manufactured  to  hold  device,  so  when  not  in  use 
hands  could  be  free  for  weapon;  clearer  resolution  on  some  map  features; 
notification  of  received  message 

(505)  no  big  glitches;  makes  navigation/communication  tasks  a  little  easier 

(506)  more  effective  if  attached  to  forearm  or  hand  to  allow  simultaneous  weapon 
handling 


RPDA  B  -  Comments  from  Handheld  Device  Evaluations 

6.  Satisfaction  With  Text  Messages  Sent 

(501)  4-  easy  to  hit  wrong  key,  wasting  time 

(502)  1-  hard  to  hit  correct  letters  because  of  irresponsiveness  of  stylus 
(505)  4-  had  to  push  down  hard  with  stylus,  which  led  to  double  typing  a  letter 

8.  Satisfaction  With  Text  Messages  Received 

No  comments 

17.  Satisfaction  With  Mapping  Functions 

(501)  2-  hard  to  read  names  on  map 

(502)  3-  hard  to  pull  up  coordinates  because  of  stylus 

18.  Satisfaction  With  SA  Tools 

No  comments 

20.  Satisfaction  With  Stylus 

(501)  4-  stylus  was  fairly  capable 

(502)  1-  horrible  responsiveness,  cumbersome  on  every  level,  especially  when 
using  keypad 

(505)  4-  had  to  push  hard 

26.  Ability  to  Use  Personal  Weapon 

(501)  1-  could  not  aim  or  shoot  weapon  while  operating  RPDA,  in  terms  of 

mechanisms  and  concentration 

(505)  3-  hard  to  operate  simultaneously 
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28.  Overall  Satisfaction  With  Reliability 

(501 )  4-  easy  to  hit  wrong  key  while  typing 

(502)  1-  stylus 

30.  Overall  Satisfaction  With  System 

(501)  4-  liked  size;  liked  receive  message  function  and  reading  old  messages, 
both  sent  and  received 

(502)  1-  stylus 

31.  Impression  of  Ease  of  Learning  and  Use 

(502)  5-  easy  to  learn 

32.  Free  Comments 

(501)  hard  to  type  and  concentrate  on  event  in  the  scenario;  system  better  for 
reconnaissance  or  sniper  mission  to  quietly  give  reports/receive  orders  w/o  radio 
noise;  in  high  speed  situations,  a  radio  is  preferable;  finding  grid  coordinates, 
distance,  direction,  of  location  in  reference  to  oneself  works  well,  but  it  would  be 
better  if  one  could  also  find  distance,  direction  between  two  points  independently 
of  one's  own  location 

(502)  too  bulky  and  stylus  too  difficult  to  use 

(505)  only  flaw  was  stylus — pushing  down  harder  made  it  difficult 

RPDA  D  -  Comments  from  Handheld  Device  Evaluations 

6.  Satisfaction  With  Text  Messages  Sent 

(528)  5-  used  chat  slang  for  brevity 

8.  Satisfaction  With  Text  Messages  Received 

No  comments 

17.  Satisfaction  With  Mapping  Functions 

No  comments 

18.  Satisfaction  With  SA  Tools 

(528)  4-  would  like  to  see  current  (?)  data  displayed — panic  button  for  key  data 

20.  Satisfaction  With  Stylus 

(528)  4-  alternate  means  if  lost?  damage  to  screen  potential 

26.  Ability  to  Use  Personal  Weapon 

(515)  2-  requires  weapon  to  be  slung  to  operate 

(527)  N/A-  M9  or  M4  more  suited  for  use  with  this  device 

(528)  none-  difficult  w/o  a  drop  capability 
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28.  Overall  Satisfaction  With  Reliability 

(527)  4-  comms  piece  came  loose  at  first  (system  crashed) 

30.  Overall  Satisfaction  With  System 

(528)  4-  software  should  be  upgraded 

31.  Impression  of  Ease  of  Learning  and  Use 

(528)  5-  would  use  in  combat  with  suggestions  made 

32.  Free  Comments 

No  comments 
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SURVEY  DATA:  INDIVIDUAL  NUMERICAL  RESPONSES 


TES1 

TES2 

TES3 

TES4 

TES5 

TES6 

TES7 

TESe 

TES9 

TES10 

TES11 

TES12 

TES13 

TES14 

TES15 

TES16 

TES17 

TES1S 

TES19 

TES201 

TES2D2 

TES2D3 

TES2D.4 

TES20.5 


TE321 

TES22 

TES23 

TES24 

TES25 

TES26 

TES27 

TES2S 

TES29 

TES3D 

TES31 

TES32 

TES33 

TES34 

TES35 

TES36 

TES37 

TES3S 

TES39 

TES40 

TES41 

TES42 

TES43 

TES44 

TES45a 

TES45b 

TES45C 

TES45d 

TES45e 

TES45f 

TES45g 

TES45h 


RPDA-D 

listed  in  order  tested 

„> 

Participant  Number 

0513 

0514 

0515 

0528 

0527 

have  cell  phone  (yes/no) 

1 

1 

0 

1 

0 

talk  oo  phone  (hours) 

9 

5 

0 

5 

0 

use  text  messaging  (hours) 

2 

2 

0 

1 

0 

customize  cell  phone  (hours) 

1 

1 

0 

0.2 

0 

play  games  or  cell  phone  (hours) 

1 

2 

0 

0 

0 

phone  numbers  stored  in  cell  phone  [#| 

40 

75 

0 

100 

0 

PDA  use 

2 

2 

2 

2 

1 

PDA  stylus  use 

3 

3 

3 

3 

1 

video  game  play  (hours) 

7 

7 

2 

0 

0 

oriine  video  games  (yes/no) 

□ 

1 

1 

0 

0 

home  computer  video  games  (yes/no) 

1 

1 

1 

0 

0 

video  game  system  video  games  (yes/no) 

1 

1 

1 

0 

0 

list  favorite  games 

* 

m 

* 

* 

has  work  computer  for  own  use  (yes/no) 

0 

0 

0 

1 

1 

has  shared  work  computer  (yes/noj 

1 

1 

0 

0 

0 

almost  never  use  computer  at  work  (yes/no) 

□ 

0 

1 

0 

0 

use  computer  in  living  quarters  (yes/no) 

1 

1 

1 

1 

1 

use  public  computers  (yes/no) 

1 

a 

1 

0 

0 

use  several  programs  at  same  lime  (yes/no) 

1 

i 

1 

1 

1 

computer  problem:  a.  slop 

5 

5 

2 

4 

5 

computer  problem:  b.  read  manual 

3 

3 

5 

5 

4 

computer  problem:  c.  read  help 

2 

2 

3 

3 

2 

computer  problem:  d  ask  tor  help 

4 

4 

4 

2 

3 

computer  problem:  e  keep  trying 

1 

1 

1 

1 

1 

RPDA-D 

listed  in  order  tested 

-> 

Participant  Number 

0513 

0514 

0515 

0528 

0527 

use  a  computer 
'use  email  (hrs  per  week) 

5 

200 

20 

200 

100 

'use  email  attachments  [hr$  per  week) 

1 

'00 

2 

50 

SO 

'use  online  Chat  or  IM  (hrs  per  week) 

1 

1000 

0 

0 

0 

'use  internet  to  find  information  (hrs  per  week) 

1 

50 

15 

50 

20 

'shop  online  (hrs  per  MONTH) 

0 

5 

1 

2 

0 

'use  online  banking  (hrs  per  MONTH) 

1 

20 

0 

2 

3 

trust  computers  (yes/no) 

1 

1 

0 

1 

1 

experience  with  radio 

2 

1 

1 

4 

4 

experience  with  GPS 

1 

1 

1 

5 

4 

experience  with  topographical  map 

3 

1 

2 

5 

4 

experience  with  military  compass 

2 

4 

2 

5 

4 

experience  with  FBCB2  or  BFT 

0 

0 

0 

4 

0 

experience  with  Tacticomp 

0 

0 

0 

0 

a 

experience  with  CDA  or  other  RPDA 

□ 

0 

0 

5 

0 

age 

19 

18 

19 

31 

27 

gender 

M 

M 

M 

M 

M 

pay  grade 

B2 

E2 

E3 

03 

E6 

branch  of  service 

USMC  USMC  USMC 

USA 

USA 

years  enlisted  service 

1 

1 

1 

0 

9 

years  commissioned  service 

0 

0 

0 

9 

0 

designator,  MOS,  or  rating 

2600 

2600 

2670 

35D 

93G 

level  of  edO cation  (1-5) 

2 

2 

3 

4 

3 

current  assignment 

* 

* 

* 

* 

* 

special  schools:  Ranger 

□ 

0 

0 

0 

0 

special  schools:  Pathfinder 

□ 

0 

0 

0 

0 

special  sc  hods:  E!B 

0 

0 

0 

0 

0 

special  sc  hods:  Air  Assault 

0 

0 

0 

0 

1 

special  sc  hods:  NCO  Course 

0 

a 

0 

0 

1 

special  sc  hods:  5  NCO  Course 

0 

0 

0 

0 

0 

special  schods:  Computer  Training 

□ 

0 

0 

0 

0 

special  schods:  HALO 

0 

0 

0 

0 

0 
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Takagi  Thesis  Raw  Data 


Participant  Number 

RPDA-D 

listed  in  order  tested 
0513  0514  0515 

0528 

0527 

Device  used  this  trial 

4 

4 

4 

4 

4 

Order 

1 

1 

1 

1 

1 

HDE1 

location 

HDE2 

(fete 

HDE3 

description  of  scenario 

HDE4 

description  of  features 

HDE5a 

#  text  messages  sent  (attempted) 

3 

3 

4 

5 

0 

HDE5b 

#  text  messages  sent  (successfil) 

3 

3 

3 

5 

0 

HDE6 

satisfaction  with  text  messages  sent 

4 

5 

4 

5 

5 

HDEZa 

#  text  messages  received  (attempted) 

3 

3 

3 

3 

3 

HDE7b 

#  text  messages  received  (successful} 

3 

3 

3 

3 

3 

HOES 

satisfaction  wilh  text  messages  received 

5 

6 

0 

5 

0 

HDE9 

satisfaction  with  comm  and  network 

4 

5 

4 

5 

4 

HDE10 

satisfaction  with  reliability  of  command  network 

S 

0 

4 

5 

0 

HDE11 

satisfaction  with  still  images 

3 

3 

4 

4 

5 

H  DEI  2a  #  rawing  videos  viewed  (atte  mpted) 

6 

6 

0 

0 

0 

H DEI 2b  #  rawing  videos  viewed  (successful) 

6 

6 

0 

0 

0 

HDE13 

satisfaction  with  quality  of  moving  video 

S 

0 

0 

0 

0 

HDE1 4a 

#  audio  tiles  listened  to  (attempted) 

6 

6 

0 

0 

0 

H  DEI  4b 

#  audio  tiles  listened  to  (successful) 

6 

6 

0 

0 

0 

HDE15 

satisfaction  with  quality  of  audo  files 

5 

5 

0 

0 

0 

H  DEI  6a 

#  of  uses  of  mapping  functions  [attempted) 

5 

5 

3 

3 

3 

H  DEI  6b  #  of  uses  of  mappi  ng  fu  nc  lions  [successftf ) 

5 

5 

3 

3 

3 

HDE17 

satisfaction  with  mapping  functions 

4 

4 

4 

4 

5 

HDE18 

satisfaction  with  SA  tools 

3 

3 

0 

4 

0 

H  DEI  9a 

#  of  uses  of  stylus  (attempted) 

10 

10 

7 

10 

20 

H  DEI  9b  #  of  uses  of  styl  us  (success!  d ) 

10 

10 

7 

10 

20 

HDE20 

satisfaction  with  stylus 

3 

4 

4 

4 

5 

HDE21a 

#of  uses  of  buttons  (attempted) 

0 

0 

0 

0 

0 

HDE21b 

#of  uses  of  buttons  (successful) 

0 

0 

0 

0 

0 

HDE22 

satisfaction  with  buttons 

0 

0 

0 

0 

0 

HDE23 

satisfaction  with  one  handed  operation 

0 

0 

0 

2 

0 

HDE24 

satisfaction  with  removing  device  to  use  it 

0 

0 

0 

0 

0 

HDE25 

satisfaction  with  stowing  device 

0 

0 

0 

0 

0 

HDE26 

ability  to  use  personal  weapon 

2 

0 

2 

0 

0 

HDE27 

ability  to  read  the  screen 

0 

0 

0 

0 

0 

HDE28 

overall  satisfaction  with  reliability 

4 

4 

0 

0 

4 

HDE29 

overall  satisfaction  with  ruggedness 

3 

3 

0 

4 

0 

HDE30 

overall  satisfaction  with  system 

4 

4 

4 

4 

5 

HDE31 

impression  of  ease  of  learning  and  use 

3 

4 

4 

5 

5 

HDE32 

free  comments 
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RRDA-A 


Participant  Number 

0502 

0506 

0501 

0503 

0505 

0504 

TES1 

have  cell  phone  (yes/no) 

0 

1 

0 

1 

1 

1 

TES2 

talk  or  phone  (hours) 

0 

7 

0 

14 

6 

S 

TES3 

use  text  messaging  (hours) 

0 

0 

0 

2 

2 

0 

TES4 

customize  cell  phone  (hours) 

0 

0 

0 

1 

1 

1 

TES5 

play  games  or  cell  phone  (hours) 

0 

1 

0 

0 

1 

0 

TES6 

phone  numbers  stored  in  cell  phone  (#) 

0 

40 

0 

70 

30 

20 

TES7 

PDA  use 

1 

2 

1 

2 

1 

1 

TES8 

PDA  stylus  use 

1 

3 

1 

3 

1 

1 

TES9 

video  game  play  (hours) 

10 

0 

6 

50 

2 

3 

TES10 

online  video  games  (yes/oo) 

0 

G 

0 

1 

1 

1 

TES11 

home  computer  video  games  (yes/no) 

0 

0 

1 

1 

1 

1 

TES12 

video  game  system  video  games  (yes/noj 

1 

1 

1 

1 

1 

0 

TES13 

list  favorite  games 

* 

* 

* 

* 

* 

* 

TES1 4 

has  work  computer  for  own  use  (yes/no) 

0 

1 

0 

1 

1 

0 

TES15 

has  shared  work  computer  (yesAno) 

0 

0 

1 

1 

1 

0 

TES16 

almost  never  use  computer  at  work  (yes/no) 

0 

0 

0 

0 

1 

TES17 

use  computer  in  living  quarters  (yes/no) 

1 

0 

0 

0 

1 

TES18 

use  public  computers  (yes/no) 

1 

1 

1 

1 

1 

TES19 

use  several  programs  at  same  lime  (yes/roo) 

0 

1 

1 

1 

1 

TES20 1  co  mpu  ter  prod  e  m :  a .  stop 

4 

4 

5 

5 

4 

TES20.2  computer  problem:  b.  read  manual 

5 

5 

4 

3 

0 

TES20. 3  co  mpu  ter  prod  e  m :  c .  read  hdp 

3 

2 

1 

2 

3 

TES20. 4  computer  problem:  d.  ask  for  hdp 

2 

3 

3 

2 

4 

1 

TES20  5  computer  problem:  e.  keep  trying 

1 

1 

2 

1 

1 

2 

Participant  Number 

RPDA-A 

0502  0506 

0501 

0503 

05O5 

0504 

TE321 

use  a  computer 

4 

TES22 

'use  email  (hre  per  week) 

1 

10 

2 

140 

20 

20 

TES23 

'use  email  attachments  (hrs  per  week) 

□ 

1 

0 

50 

10 

5 

TES24 

'use  online  chat  or  IM  (hrs  per  week) 

□ 

0 

0 

7 

15 

5 

TES25 

'use  internet  to  find  information  (hrs  per  week) 

1 

70 

1 

7 

20 

20 

TES26 

'shop  online  (hrs  per  MONTH) 

0 

4 

3 

4 

3 

0 

TES27 

'use  online  banking  [hrs  per  MONTH) 

0 

2 

0 

2 

5 

10 

TES2S 

trust  computers  (yes/no) 

1 

1 

1 

1 

1 

1 

TES29 

experience  with  radio 

1 

1 

1 

0 

1 

0 

TES30 

experience  with  GPS 

□ 

0 

1 

0 

1 

0 

TES31 

experience  with  topographical  map 

3 

1 

2 

3 

2 

1 

TES32 

experience  with  military  compass 

3 

1 

2 

3 

2 

1 

TES33 

experience  with  FBCB2  or  BFT 

0 

0 

1 

0 

G 

0 

TES34 

experience  with  Tacticomp 

0 

1 

1 

0 

0 

0 

TES35 

experience  with  CDA  or  other  RPDA 

□ 

G 

1 

0 

0 

0 

TES36 

age 

24 

36 

20 

IB 

19 

21 

TES37 

gender 

M 

M 

M 

M 

M 

F 

TES3S 

pay  grade 

E4 

E4 

E3 

E2 

E2 

E3 

TES39 

branch  d  service 

USA 

USA 

USA 

USA 

USA 

USA 

TES4D 

years  enlisted  service 

1 

1 

1 

1 

1 

1 

TES41 

years  commissioned  service 

0 

0 

0 

0 

G 

0 

TES42 

designator,  MOS,  or  rating 

98X 

97  E 

98 

98G 

98X 

98  G 

TES43 

level  of  education  (1-5) 

4 

4 

3 

2 

3 

3 

TES44 

current  assignment 

* 

* 

to 

TES45a 

special  schools:  Ranger 

0 

0 

0 

0 

0 

0 

TES45b 

special  schools:  Pathfinder 

0 

0 

0 

0 

0 

0 

TES45c 

special  schools:  EIB 

□ 

0 

0 

0 

0 

0 

TES45d 

special  schools:  Air  Assault 

0 

0 

0 

0 

0 

0 

TES45e 

special  schools:  NOO  Course 

0 

G 

0 

0 

0 

0 

TES45f 

special  schools:  SNOO  Course 

□ 

0 

0 

0 

0 

0 

TES45g 

special  schools:  Computer  Training 

□ 

0 

0 

0 

0 

0 

TES45h 

special  schools:  HALO 

□ 

0 

0 

0 

0 

0 
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Takagi  Thesis  Raw  Data 


Participant  Number 

RPDA-A 

0502  0506 

□501 

0503 

0505 

0504 

Device  used  this  trial 

1 

1 

1 

1 

1 

1 

Order 

1 

1 

2 

2 

1 

2 

HDE1 

location 

HDE2 

(fete 

HDE3 

description  of  scenario 

HDE4 

description  of  features 

HDE5a 

#  text  messages  sent  (attempted) 

5 

2 

5 

4 

HDE5b 

#  text  messages  sent  (successfii) 

s 

2 

0 

4 

HDE6 

satisfaction  with  text  messages  sent 

4 

4 

4 

4 

HDEZa 

#  text  messages  received  (attempted) 

5 

3 

5 

4 

HDE7b 

#  text  messages  received  (successful} 

5 

3 

5 

4 

HOES 

satisfaction  wilh  text  messages  received 

5 

5 

4 

5 

HDE9 

satisfaction  with  comm  and  network 

4 

4 

4 

4 

HDEIO 

satisfaction  with  reliability  of  command  network 

5 

2 

4 

4 

HDE11 

satisfaction  with  still  images 

A 

4 

1 

3 

H  DEI  2a  #  rawing  videos  viewed  (atte  mpted) 

1 

6 

0 

6 

H DEI 2b  #  rawing  videos  viewed  (successful) 

1 

6 

0 

6 

HDE13 

satisfaction  with  quality  of  moving  video 

5 

5 

0 

0 

HDE1 4a 

#  audio  tiles  listened  to  (attempted) 

1 

7 

0 

6 

H  DEI  4b 

#  audio  tiles  listened  to  (successful) 

1 

7 

0 

6 

HDE15 

satisfaction  with  quality  of  audo  files 

5 

5 

0 

5 

H  DEI  6a 

#  of  uses  of  mapping  functions  [attempted) 

3 

4 

3 

5 

H  DEI  6b  #  of  uses  of  mappi  ng  fu  nc  lions  [successfii ) 

3 

4 

3 

5 

HDE17 

satisfaction  with  mapping  functions 

4 

2 

3 

4 

HDE18 

satisfaction  with  SA  fools 

□ 

3 

4 

5 

H  DEI  9a 

#  of  uses  of  stylus  (attempted) 

75 

7 

8 

10 

H  DEI  9b  #  of  uses  of  styl  us  (success!  d ) 

75 

7 

8 

10 

HDE20 

satisfaction  with  stylus 

5 

4 

1 

5 

HDE21a 

#of  uses  of  buttons  (attempted) 

0 

0 

0 

1 

HDE21b 

#of  uses  of  buttons  (successful) 

□ 

0 

0 

1 

HDE22 

satisfaction  with  buttons 

□ 

0 

0 

3 

HDE23 

satisfaction  with  one  handed  operation 

2 

3 

1 

5 

HDE24 

satisfaction  with  removing  device  to  use  if 

0 

2 

0 

3 

HDE25 

satisfaction  with  stowing  device 

0 

0 

3 

3 

HDE26 

ability  to  use  personal  weapon 

1 

2 

1 

2 

HDE27 

ability  to  read  the  screen 

5 

0 

0 

4 

HDE28 

overall  satisfaction  with  reliability 

5 

6 

2 

5 

HDE29 

overall  satisfaction  with  ruggedness 

5 

0 

0 

5 

HDE30 

overall  satisfaction  with  system 

5 

4 

2 

5 

HDE31 

impression  of  ease  of  learning  and  use 

5 

5 

5 

5 

HDE32 

free  comments 

* 

irk 

* 

* 
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Takagi  Thesis  Raw  Data 


Participant  Number 

RPDA-B 

0502  0506 

0501 

0503 

0505 

0504 

Device  used  this  trial 

2  2 

2 

2 

2 

2 

Order 

2  2 

1 

1 

2 

1 

HDE1 

location 

HDE2 

(fete 

HDE3 

description  of  scenario 

HDE4 

description  of  features 

HDE5a 

#  text  messages  sent  (attempted) 

1 

6 

1 

HDE5b 

#  text  messages  sent  (successfii) 

1 

6 

1 

HDE6 

satisfaction  with  text  messages  sent 

1 

4 

4 

HDEZa 

#  text  messages  received  (attempted) 

1 

6 

1 

HDE7b 

#  text  messages  received  (successful} 

1 

6 

1 

HOES 

satisfaction  wilh  text  messages  received 

5 

4 

a 

HDE9 

satisfaction  with  comm  and  network 

2 

4 

A 

HDEIO 

satisfaction  with  reliability  of  command  network 

S 

5 

a 

HDE11 

satisfaction  with  still  images 

3 

2 

4 

H  DEI  2a  #  rawing  videos  viewed  (atte  mpted) 

0 

0 

1 

H DEI 2b  #  rawing  videos  viewed  (successful) 

0 

0 

1 

HDE13 

satisfaction  with  quality  of  moving  video 

□ 

0 

A 

HDE1 4a 

#  audio  tiles  listened  to  (attempted) 

□ 

0 

1 

H  DEI  4b 

#  audio  tiles  listened  to  (successful) 

□ 

0 

1 

HDE15 

satisfaction  with  quality  of  audo  files 

0 

0 

4 

H  DEI  6a 

#  of  uses  of  mapping  functions  [attempted) 

3 

3 

2 

H  DEI  6b  #  of  uses  of  mappi  ng  fu  nc  lions  [successfii ) 

3 

3 

2 

HDE17 

satisfaction  with  mapping  functions 

3 

2 

A 

HDE18 

satisfaction  with  SA  fools 

□ 

3 

5 

H  DEI  9a 

#  of  uses  of  stylus  (attempted) 

50 

7 

3 

H  DEI  9b  #  of  uses  of  styl  us  (success!  d ) 

10 

7 

3 

HDE20 

satisfaction  with  stylus 

1 

4 

4 

HDE21a 

#of  uses  of  buttons  (attempted) 

0 

0 

0 

HDE21b 

#of  uses  of  buttons  (successful) 

0 

0 

0 

HDE22 

satisfaction  with  buttons 

0 

0 

0 

HDE23 

satisfaction  with  one  handed  operation 

0 

1 

3 

HDE24 

satisfaction  with  removing  device  to  use  if 

0 

3 

0 

HDE25 

satisfaction  with  stowing  device 

0 

3 

0 

HDE26 

ability  to  use  personal  weapon 

0 

1 

3 

HDE27 

ability  to  read  the  screen 

0 

5 

A 

HDE28 

overall  satisfaction  with  reliability 

1 

4 

5 

HDE29 

overall  satisfaction  with  ruggedness 

0 

0 

5 

HDE30 

overall  satisfaction  with  system 

1 

4 

5 

HDE31 

impression  of  ease  of  learning  and  use 

5 

5 

a 

HDE32 

free  comments 

* 

** 

* 
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No  Device 


Participant  Number 

0507 

0517 

0518 

TES1 

have  cell  phone  (yes/no) 

□ 

1 

1 

TES2 

talk  on  phone  (hours) 

□ 

6 

5 

TES3 

use  text  messaging  (hours) 

0 

1 

0 

TES4 

customize  cdl  phone  (hours) 

0 

1 

0 

TES5 

play  games  on  cdl  phone  (hours) 

0 

G 

0 

TES6 

phone  numbers  stored  in  cell  phone  [#) 

0 

40 

20 

TE57 

PDA  use 

2 

1 

1 

TES8 

PDA  stylus  use 

4 

1 

1 

TES9 

video  game  play  (hours) 

5 

8 

3 

TES10 

online  video  games  (yes/no) 

1 

0 

0 

TES11 

home  computer  video  games  (yes/no) 

1 

0 

0 

TES12 

video  game  system  video  games  (yes/no) 

0 

1 

1 

TES13 

list  favorite  games 

* 

A 

* 

TES14 

has  work  computer  for  own  use  (yes/no) 

0 

c 

1 

TES15 

has  shared  work  computer  (yes/no) 

1 

1 

0 

TES16 

almost  never  use  computer  at  work  (yes/no) 

1 

1 

0 

TES17 

use  computer  in  living  quarters  (yes/no) 

0 

0 

0 

TES1S 

use  public  computers  (yes/no) 

1 

1 

0 

TES19 

use  several  programs  aE  same  lime  (yes/no) 

1 

1 

1 

TES201 

TES20.2 

TES203 

TES20.4 

TES20.5 

computer  problem:  a.  stop 
computer  problem:  b.  read  manual 
computer  problem:  c.  read  hdp 
computer  problem:  d.  ask  for  hdp 
computer  problem:  e.  keep  trying 

1 

1 

0 

4 

3 

2 

1 

Participant  Number 

No  Device 

0507  0517 

0518  : 

TES21 

TES22 

use  a  computer 
'use  email  (hrs  per  week) 

35 

4 

100 

TES23 

'use  email  attachments  [hrs  per  week) 

5 

2 

20 

TES24 

'use  online  chat  or  IM  (hrs  per  week) 

0 

1 

0 

TES25 

'use  internet  to  find  informalion  (hrs  per  week) 

7 

7 

50 

TES26 

'shop  online  (hrs  per  MONTH) 

0 

4 

2 

TES27 

'use  online  banking  [hrs  per  MONTH) 

2 

2 

0 

TES2S 

trust  computers  (yes/no) 

1 

1 

0 

TES23 

experience  with  radio 

1 

1 

3 

TES3D 

experience  with  GPS 

0 

1 

4 

TES31 

experience  with  topographical  map 

2 

2 

5 

TES32 

experience  with  military  compass 

3 

2 

5 

TES33 

experience  with  FBCB2  or  BFT 

0 

1 

0 

TES34 

experience  with  Tacticomp 

0 

1 

0 

TES35 

experience  with  CDA  or  other  RFDA 

0 

1 

0 

TES36 

ago 

18 

22 

35 

TES37 

gender 

M 

M 

M 

TES38 

pay  grade 

E2 

E4 

E5 

TES39 

branch  of  service 

USA 

USMC 

USA 

TES4II 

years  enlisted  service 

1 

2 

9 

TES41 

years  commissioned  service 

0 

0 

0 

TES42 

designator,  MOS,  or  rating 

1 ET 

2600 

11B2P 

TES43 

level  of  education  (1-5) 

2 

3 

2 

TES44 

current  assignment 

* 

A 

* 

TES45a 

special  sc  hods  :  Ranger 

0 

0 

0 

TES45b 

special  sc  hods:  Pathfinder 

0 

0 

0 

TES45c 

special  sc  hods:  EIB 

0 

0 

0 

TES45d 

special  sc  hods:  Air  Assault 

0 

0 

0 

TES45e 

special  sc  hods:  NGO  Course 

0 

G 

1 

TES45f 

special  schods:  5NOO  Course 

0 

0 

0 

TES45g 

special  schods:  Computer  Training 

0 

0 

0 

TES45h 

special  schods:  HALO 

0 

0 

0 
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4. 


SURVEY  DATA:  AGGREGATE  SUMMARY  OF  RESPONSES 


Takagi  Thesis  Survey  Aggregate  Results 


H  =  80%  or  more 

Overall 

na 

"  50%  to  80% 

nla 

no 

yes 

Eval 

Category 

0 

1 

2 

3 

4 

5 

Total 

TES 

1 

have  cell  phone  (yes/no) 

36% 

mm 

1 

14 

TES 

2 

talk  on  phone  [hours) 

median= 

5.00 

mean= 

4.69 

sd= 

409 

14 

TES 

3 

use  text  messaging  (hours) 

median= 

mean= 

0.77 

sd= 

093 

14 

TES 

4 

customize  cell  phone  {hours) 

median= 

0.20 

mean= 

0.48 

sd= 

0.51 

14 

TES 

5 

play  games  on  cell  phone  (hours) 

median= 

mean= 

0.38 

sd= 

065 

14 

TES 

6 

phone  numbers  stored  in  cell  phone  (#) 

median= 

30.00 

mean= 

33.46 

sd= 

32  24 

14 

TES 

7 

PDA  use 

i 

50% 

14 

TES 

8 

PDA  stylus  use 

^'.Vl  50% 

H“ 

43% 

7% 

14 

TES 

9 

video  game  play  (hours) 

median= 

5.00 

mean= 

7.92 

sd= 

13.02 

14 

TES 

10 

online  video  games  (yes/no) 

43% 

14 

TES 

11 

home  computer  video  games  (yes/no) 

43% 

57% 

14 

TES 

12 

video  game  system  video  games  (yes/ro) 

29% 

71% 

14 

TES 

13 

list  favorite  games 

13 

TES 

14 

has  work  computer  for  own  use  (yes/no) 

57%  | 

43% 

14 

TES 

15 

has  shared  work  computer  (yes/no) 

50% 

50% 

I 

14 

TES 

16 

almost  never  use  computer  at  work  (yes/no) 

69%  1 

31% 

13 

TES 

17 

use  computer  in  living  quarters  (yes/no) 

46% 

54% 

13 

TES 

18 

use  public  computers  (yes/no) 

3f% 

63% 

13 

TES 

19 

use  several  programs  at  same  time  (yes/no) 

S%  | 

92*4 

13 

TES 

20.1 

computer  problem:  a.  stop 

9% 

36% 

■SI 

11 

TES 

202 

computer  problem:  b.  read  manual 

27% 

27% 

45% 

11 

TES 

20.3 

computer  problem:  c.  read  help 

9% 

45% 

45% 

11 

TES 

204 

computer  problem:  d.  ask  for  help 

8% 

33% 

25% 

33% 

12 

TES 

205 

conpjter  problem:  e.  Reqp  trying 

|  14% 

14 

21 

use  a  computer 

median= 

4.00 

mean= 

4.00 

sd= 

#DIWffl 

1 

TES 

22 

use  email  (hrs  per  week) 

median= 

20.00 

mean= 

58.23 

sd= 

75  33 

14 

TES 

23 

use  email  attachments  (hrs  per  week) 

median= 

5.00 

mean= 

18.92 

sd= 

30.15 

14 

TES 

24 

use  online  chat  or  IM  (hrs  per  week) 

median= 

mean= 

79.15 

sd= 

276.71 

14 

TES 

25 

use  internet  to  find  information  (hrs  per  week) 

median^ 

15.00 

mean= 

23.00 

sd= 

2363 

14 

TES 

26 

shop  online  [hrs  per  MONTH) 

median^ 

2.00 

mean= 

2.15 

sd= 

1.82 

14 

TES 

27 

use  online  banking  [hrs  per  MONTH) 

median^ 

2.00 

mean= 

3.54 

sd= 

565 

14 

TES 

28 

trust  computers  w/  personal  info  [yes/no) 

14%  1 

86% 

1 

14 

TES 

29 

experience  with  radio 

14% 

7% 

7% 

14% 

14 

TES 

30 

experience  with  GPS 

36% 

43% 

14% 

7% 

14 

TES 

31 

experience  with  topographical  map 

21% 

36% 

21% 

7% 

14% 

14 

TES 

32 

experience  with  military  compass 

14% 

36% 

21% 

14% 

14% 

14 

TES 

33 

experience  with  FBCB2  or  BFT 

14% 

7% 

14 

TES 

34 

experience  with  Tacticomp 

21% 

14 

TES 

35 

experience  with  CDA  or  other  RPDA 

14% 

7% 

14 
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Takagi  Thesis  Survey  Aggregate  Results 


BlU  =  80%  or  more  Overall 


Evil 

=  50%  to  80% 

Category 

nfa 

no  yes 

0  1 

2 

3 

4 

5 

Total 

TES 

36 

age 

median=  20.00 

mean= 

23.08 

sd= 

655 

14 

TES 

37 

gender 

14 

TES 

38 

pay  grade 

14 

TES 

39 

branch  of  service 

14 

TES 

40 

years  enlisted  service 

median=  1 .00 

mean= 

162 

sd= 

226 

14 

TES 

41 

years  commissioned  servcie 

median= 

mean= 

0.75 

sd= 

260 

14 

TES 

42 

designator.  MOS.  or  rating 

14 

TES 

43 

level  of  education  (1-5) 

36% 

43% 

21% 

14 

TES 

44 

current  assignment 

11 

TES 

45a 

special  sc  hods:  Ranger 

14 

TES 

45b 

special  sc  hod  s :  Pathfi  nder 

14 

TES 

45c 

special  sc  hod  s:  El  B 

14 

TES 

45d 

special  sc  hod  s :  Ai  r  Assa  ul  t 

7% 

14 

TES 

45e 

special  sc  hod  s :  NCO  Course 

1  14% 

14 

TES 

45f 

special  sc  hod  s :  SNC0  Con  rse 

14 

TES 

45g 

special  sc  hod  s :  Compu  ter  I rai  ni  ng 

14 

TES 

45h 

special  sc  hods:  HALO 

14 
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Takagi  Thesis  Survey  Aggregate  Results 


Eval 


=  80%  or  more 
=  50%  to  80% 

Category 


Overall 

nfa 

no 

0 


yes 

1 


Total 


HDE 

1  1 

HDE 

2  - 

HDE 

3  r 

HDE 

4  < 

HDE 

5a  i 

HDE 

5b  ! 

HDE 

6  ! 

HDE 

7a  i 

HDE 

7b  \ 

HDE 

8  : 

HDE 

9  : 

HDE 

10  ! 

HDE 

11  ! 

HDE 

12a 

HDE 

12b 

HDE 

13 

HDE 

14a 

HDE 

14b 

HDE 

15 

HD£ 

16a  i 

HDE 

16b  i 

HDE 

17  ! 

HDE 

18 

HDE 

19a  i 

HDE 

19b  i 

HDE 

20  ! 

HDE 

21a  3 

HDE 

21b  3 

HDE 

22  : 

HDE 

23  : 

HDE 

24  ! 

HDE 

25  ! 

HDE 

26  i 

HDE 

27  i 

HDE 

28  i 

HDE 

29  i 

HDE 

30  i 

HD1 

31  i 

HD1 

32  1 

Device  used  this  trial 
Device  used  previously 
location 
date 

description  of  scenario 
description  of  features 


satisfaction  v 
#  text  messages  received  (attempted) 


satisfaction  with  text  messages  received 
satisfaction  with  comm  and  network 
10  sa  tisfactio  n  wit  h  rel  lability  of  comm  an  d  network 
satisfaction  with  still  images 


35% 


35% 


29% 


8% 


8% 

8% 

8% 


0.50  mean= 
0.50  mean= 


0.50 

0.50 


mean= 

mean= 


33% 

2.17 

2.17 

2-26 

2.26 


25% 

42% 

sd= 

sd= 

8% 

sd= 

sd= 

8% 


free  comments 


17 

17 

0 

0 

0 

0 


median= 

4.00 

mean= 

3.67 

sd= 

1.67 

12 

median= 

3.60 

mean= 

3.58 

sd= 

1.68 

12 

8% 

Ha 

25% 

12 

median= 

3.00 

mean= 

3.33 

sd= 

1.50 

12 

median= 

3.00 

mean= 

3.33 

sd= 

1.50 

12 

8% 

286 

286 

42% 

2.99 

2.99 

42% 


12 

12 

12 

12 

12 

12 

12 

12 

12 

12 


#  of  uses  of  mapping  functions  (attempted) 

median= 

3.00 

meaiF 

3.50 

sd= 

1.00 

12 

#  of  uses  of  mapping  functions  (successfd) 

median= 

3.00 

mean= 

3.50 

sd= 

1.00 

12 

satisfaction  with  mapping  functions 

17% 

17% 

BB 

8% 

12 

satisfaction  with  SA  tods 

33% 

33% 

17% 

17% 

12 

#  of  uses  of  stylus  (attempted) 

median^ 

10.00 

mean= 

18.08 

sd= 

21.79 

12 

#  of  uses  of  stylus  (successfd) 

median= 

10.00 

mean= 

14.75 

sd= 

19.39 

12 

satisfaction  with  stylus 

17% 

3% 

E9 

25% 

12 

#  of  uses  of  buttons  (attempted) 

median= 

mean= 

0.08 

sd= 

029 

12 

#  of  uses  of  buttons  (successful) 

median^ 

mean= 

0.08 

sd= 

029 

12 

satisfaction  with  buttons 

92% 

e% 

12 

satisfaction  with  one  handed  operation 

I  42% 

17% 

17% 

17% 

8% 

12 

satisfaction  with  removing  device  to  use  it 

75% 

8% 

17% 

12 

satisfaction  withstowng  device 

75% 

25% 

12 

ability  to  use  personal  weapon 

33% 

25% 

33% 

3% 

12 

ability  to  read  the  screen 

67%  1 

17% 

17% 

12 

overall  satisfaction  wth  reliability 

8% 

8% 

8% 

33% 

42% 

12 

overall  satisfaction  with  ruggedness 

1 

17% 

8% 

25% 

12 

overall  satisfaction  with  system 

8% 

8% 

I 

50*4 

33% 

12 

impression  of  ease  of  learning  and  use 

8% 

17% 

75%  § 

12 
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Takagi  Thesis  Survey  Aggregate  Results 


BlU  =  80%  or  more  RP  DA  -D 


=  50%  to  80% 

nfa 

no 

yes 

Eval 

Category 

0 

1 

2 

3 

4 

5 

Totfil 

TES 

1  have  cell  phone  (yes/ho) 

40% 

60% 

5 

TES 

2  talk  on  phone  (hours) 

median= 

5.00 

mearF 

4.75 

sd= 

369 

5 

TES 

3  use  text  messaging  (hours) 

median= 

150 

mearF 

1.25 

sd= 

096 

5 

TES 

A  customize  cd  1  phone  ( hours) 

median= 

0.60 

mearF 

0.55 

sd= 

053 

5 

TES 

5  play  games  on  cell  phone  (hours) 

median= 

0.50 

mearF 

0.75 

sd= 

0.96 

5 

TES 

6  phone  numbers  stored  in  cell  phone  (#) 

median= 

57.50 

meanr 

53.78 

sd= 

43  47 

5 

TES 

7  PDA  use 

20% 

B;[iM 

5 

TES 

8  PDA  stylus  use 

20% 

\ 

80% 

l 

5 

TES 

9  video  game  play  (hours)  | 

medtan= 

4.50 

mean= 

4.00 

sd= 

356 

5 

TES 

10  oriine  video  games  (y  es/no) 

mm 

40% 

5 

TES 

1 1  h  ome  computer  video  games  (yes/no) 

40% 

6 

TES 

1 2  video  ga  me  system  video  games  (yes/no) 

40% 

5 

TES 

1 3  list  favorite  games  | 

4 

TES 

14  has  work  computer  for  own  use  (yes/no) 

60% 

40% 

5 

TES 

15  has  shared  work  computer  (yes/no) 

60% 

40% 

5 

TES 

16  al  most  never  u  se  co mputer  at  work  (yes/no) 

60% 

20% 

5 

TES 

17  use  compu  ter  i  n  IM  ng  quarters  (yes/no) 

n 

100% 

5 

TES 

18  use  public  computers  (y es/no) 

40% 

5 

TES 

1 9  u  se  several  prog  ra  ms  at  sa  me  ti  me  (yes/ no) 

1 

Mi'WM 

1 

5 

TES 

20 . 1  corrpi  ter  probl  em:  a.  stop 

20% 

20% 

60% 

5 

TES 

20 . 2  cornpu  ter  probl  em:  b.  read  ma  nu  al 

40% 

20% 

40% 

5 

TES 

20 . 3  cornpu  ter  probl  em:  c .  read  bd  p 

60% 

40% 

5 

TES 

20.4  computer  problem:  d.  ask  for  help 

20% 

20% 

5 

TES 

20 . 5  cornpu  ter  probl  em:  e.  keep  Eryi  ng 

i 

100% 

5 

21  use  a  computer 

median^ 

#NUM! 

mearF 

#DIW0! 

sd= 

#OIV/ffl 

0 

TES 

22  use  email  [hrs  per  week) 

median= 

110.00 

mearF 

106.25 

sd= 

108  43 

5 

TES 

23  use  email  attachments  (hrs  per  week) 

median= 

26.00 

mearF 

38.25 

sd= 

47.09 

5 

TES 

24  u  se  onli  ne  ch  at  or  t  M  (hrs  per  week) 

median= 

0.50 

mearF 

250.25 

sd= 

499  83 

5 

TES 

25  use  ! ntern et  to  find  i nformatron  ( hrs  per  week) 

median^ 

32.50 

mearF 

29.00 

sd= 

24.91 

5 

TES 

26  shop  online  (hrs  per  MONTH) 

median^ 

1.50 

mearF 

2.00 

sd= 

2.16 

5 

TES 

27  use  online  banki  ng  (hrs  per  MONT H) 

median^ 

1.50 

mearF 

5.75 

sd= 

954 

5 

TES 

28  trust  c  ompu  lers  w /  person  al  info  (yes/no) 

20%  | 

80% 

5 

TES 

29  experiencewithradio 

40% 

20% 

40% 

5 

TES 

30  experiencewithGPS 

1 

60% 

20% 

20% 

5 

TES 

3 1  experi  ence  with  topog  raphicat  map 

20% 

20% 

20% 

20% 

20% 

5 

TES 

32  experi  ence  with  mil  i  tary  compass 

40% 

40% 

20% 

5 

TES 

33  experi  ence  with  FBCB2  or  BFT 

80% 

20% 

5 

TES 

34  experi  ence  with  T ac  ticomp 

100% 

5 

TES 

35  experience  with  CDA  or  other  RPDA 

80% 

20% 

5 
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BlU  =  80%  or  more  RP  DA  -D 


=  50%  to  80% 

nfa 

no  yes 

Evil 

Category 

0  1 

2 

3 

4 

5 

Total 

TES 

36  age 

median=  19.00 

mean= 

2175 

sd= 

6.18 

5 

TES 

37  gender 

5 

TES 

38  pay  grade 

5 

TES 

39  branch  of  service 

5 

TES 

40  years  enlisted  service 

median=  1 .00 

mearF 

0.75 

sd= 

0.50 

5 

TES 

4 1  years  commissio  ned  servcie 

median= 

meaiF 

2.25 

sd= 

450 

5 

TES 

42  designator.  MOS.  or  rating 

5 

TES 

43  level  of  education  (1-5) 

40% 

40% 

20% 

5 

TES 

44  current  assignment 

5 

TES 

45a  special  sc  hod  s :  Ran  g  er 

5 

TES 

45  b  special  sc  hod  s :  Pathfi  nder 

5 

TES 

45c  special  sc  hods:  El  El 

5 

TES 

45d  special  sc  hod  s :  Ai  r  Assa  ul  t 

1  20% 

a 

TES 

45e  special  sc  hod  s :  NCO  Course 

1  20% 

a 

TES 

45f  special  sc  hod  s :  SNC0  Con  rse 

5 

TES 

45g  special  sc  hod  s :  Compu  ter  I rai  ni  ng 

100% 

5 

TES 

45h  special  sc  hods:  HALO 

100%  ImSm. 

5 
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B<111  =  80%  or  more 

RPDA-D 

=  50%  to  80% 

nfa 

no 

0 

Evil 

Category 

Device  used  this  trial 

Device  used  previously 

HDE 

1 

location 

HDE 

2 

date 

HDE 

3 

description  of  scenario 

HDE 

4 

description  of  features 

HDE 

5a 

#texl  messages  sent  (attempted) 

median^ 

HDE 

5b 

#  text  messages  sent  (success*  J) 

median= 

HDE 

6 

satisfaction  with  text  messages  sent 

HDE 

7a 

#texl  messages  received  (attempted) 

median= 

HDE 

7b 

#texl  messages  received  [successful) 

median= 

HDE 

8 

satisfaction  with  text  messages  received 

HDE 

9 

satisfaction  with  comm  and  network 

HDE 

10 

satisfaction  with  reliability  of  comm  and  network 

HDE 

11 

satisfaction  with  still  images 

HDE 

12a 

#  moving  videos  viewed  (attempted) 

medan= 

HDE 

12b 

#  movi  ng  vi  deos  vi  ewed  [successful ) 

median= 

HDE 

13 

satisfaction  with  quality  of  moving  video 

Bn 

HDE 

14a 

#  audi  o  til  es  I  is  tened  So  (at  te  npted) 

median^ 

HDE 

14b 

#  audi  o  til  es  lis  toned  So  (successful ) 

median^ 

HDE 

15 

satisfaction  with  quality  of  audo  files 

HDE 

16a 

#  of  uses  of  mapping  functions  (attempted) 

median= 

HDE 

16b 

#  of  uses  of  mapping  functions  (successfd) 

medan= 

HDE 

17 

satisfaction  with  mapping  functions 

HDE 

18 

satisfaction  with  £A  tods 

40% 

HDE 

19a 

#  of  uses  of  slyfus  (attempted) 

median^ 

HDE 

19b 

#  of  uses  of  stylus  (successfd) 

median= 

HDE 

20 

satisfaction  with  stylus 

HDE 

21a 

#  of  uses  of  buttons  (attempted) 

median= 

HDE 

21b 

#  of  uses  of  buttons  (successful) 

median^ 

HDE 

22 

satisfaction  with  buttons 

100% 

HDE 

23 

satisfaction  with  one  handed  operation 
satisfaction  with  removing  device  to  use  it 

HDE 

24 

100% 

HDE 

25 

satisfaction  withstowng  device 

100% 

HDE 

26 

ability  to  use  personal  weapon 

HDE 

27 

ability  to  read  the  screen 

100%  I 

HDE 

28 

overall  satisfaction  wth  reliability 

20% 

HDE 

29 

overall  satisfaction  with  ruggedness 

40% 

HDE 

30 

overall  satisfaction  with  system 

HDE 

31 

impression  of  ease  of  learning  and  use 

HDE 

32 

free  comments 

yes 

1  2  3  4  5  Total 

5 


4.00 

mean= 

4.00 

3.00 

mean= 

3.80 

3.00 

mean= 

3.00 

3.00 

mean= 

3.00 

mean= 

40% 

2.40 

mean= 

2.40 

. 

mean= 

2.40 

mean= 

2.40 

3.00 

mean= 

3.80 

3.00 

meao= 

3.80 

10.00 

mean= 

40% 

11.40 

10.00 

mean= 

1140 

mean= 

20% 

- 

mean= 

- 

20% 

40% 

40% 

20% 


5 

0 

0 

0 

0 


sd= 

100 

5 

sd= 

110 

5 

40% 

5 

sd= 

- 

5 

sd= 

- 

5 

1 

100%  1 

0 

60%  | 

40% 

0 

20%  | 

0 

40% 

20% 

5 

sd= 

329 

5 

sd= 

329 

5 

40% 

0 

sd= 

3  29 

0 

sd= 

3  29 

0 

40% 

5 

sd= 

1.10 

5 

sd= 

1.10 

5 

20% 

5 

20% 

5 

sd= 

498 

5 

sd= 

498 

5 

E9 

20% 

5 

sd= 

- 

5 

sd= 

- 

5 

5 

5 

5 

5 

5 

5 


EM 

20% 

5 

20% 

5 

80%  I 

20% 

5 

40% 

40% 

5 

0 
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MU  =  80%  or  more 
=  50%  to  80% 


RPDA-A 

n/a 

no 


Eval 


Category 


0 


yes 

1 


2 


3 


4  5  Total 


TES 

1 

TES 

2 

TES 

3 

TES 

4 

TES 

5 

TES 

6 

TES 

7 

TES 

8 

TES 

9 

TES 

10 

TES 

11 

TES 

12 

TES 

13 

TES 

14 

TES 

15 

TES 

16 

TES 

17 

TES 

IS 

TES 

19 

TES 

20.1 

TES 

202 

TES 

203 

TES 

204 

TES 

205 

21 

TES 

22 

TES 

23 

TES 

24 

TES 

25 

TES 

26 

TES 

27 

TES 

28 

TES 

29 

TES 

30 

TES 

31 

TES 

32 

TES 

33 

TES 

34 

TES 

35 

have  cell  phone  (yes/no) 
talk  on  phone  (hours) 
use  text  messaging  (hours) 
customize  cell  phone  (hours) 
ptay  games  on  cell  phone  (hours) 
phone  numbers  stored  In  cell  phone  (#) 

PDA  use 
PDA  stylus  use 
video  game  play  (hours) 
online  video  games  (yes/no) 
home  computer  video  games  (yes/no) 
video  game  system  video  games  (yes/no) 
list  favorite  games 
has  work  computer  for  own  use  (yes/no) 
has  shared  work  computer  (yes/no) 
almost  never  use  computer  at  work  (yes/no) 
use  computer  in  living  quarters  (yes/no) 
use  public  computers  (yes/no) 
use  several  programs  at  same  time  (yes/no) 
computer  problem:  a.  stop 
computer  problem:  b.  read  manual 
computer  problem:  c.  read  help 
computer  problem:  d.  ask  for  help 
computer  problem:  e.  keep  trying 
use  a  computer 
use  email  (hrs  per  week) 
use  email  attachments  (hrs  per  week) 
use  online  chat  or  IM  (hrs  per  week) 
use  internet  to  find  information  (hrs  per  week) 
shop  online  (hrs  per  MONTH) 
use  online  banking  (hrs  per  yONTH) 


33% 

67%  1 

median= 

5.00 

mean= 

5.17 

sd= 

5.19 

median= 

mean= 

0.67 

sd= 

1.03 

median= 

0.50 

mean= 

0.50 

sd= 

0.55 

median= 

mean= 

0.33 

sd= 

0.52 

median= 

25.00 

mean= 

26.67 

sd= 

26  58 

19 

33% 

33% 

medfan= 

4.50 

mean= 

11.83 

sd= 

19.02 

mm 

50% 

33% 

67% 

17% 

83% 

median^ 

#NUM! 

mean= 

#DIW01 

sd= 

#DIV/ffl 

50*/. 

50% 

80% 

20% 

40% 

100% 

20% 

eo% 

1 

80% 

40% 

20% 

20% 

1  60% 

20% 

40% 

40% 

17% 

33% 

33% 

17% 

MM 

33% 

median^ 

4.00 

mean= 

4.00 

sd= 

#OIWffl 

median= 

15.00 

mean= 

32.17 

sd= 

5347 

median= 

3.00 

mearF 

11.00 

sd= 

19.49 

median= 

250 

mean= 

4.50 

sd= 

596 

median^ 

13.50 

mean= 

19.63 

sd= 

26  04 

median^ 

3.00 

mean= 

2.33 

sd= 

1  86 

median^ 

2.00 

mean= 

3.17 

sd= 

382 

trust  computers  w i  personal  info  [yes/no) 
experience  with  radio 
experience  with  GPS 
experience  with  topographical  map 
experience  with  military  compass 
experience  with  FBCB2  or  BFT 
experience  with  Tacticomp 
experience  with  CDA  or  other  RPDA 


100% 


33% 


ww.m 

33% 

33% 

33% 

33% 

33% 

33% 

33% 

83% 

17% 

67% 

33% 

83% 

17% 

6 

6 

6 

6 

6 

6 

6 

6 

6 

6 

6 

6 

6 

6 

6 

5 

5 

5 

5 

5 

5 

5 

6 
6 
1 
6 
6 
6 
6 
6 
6 
6 
6 
6 
6 
6 
6 
6 
6 
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ESS "  80%  or  more 

=  50%  to  80%  I 

RPDA-A 

nfa 

no 

yes 

Eval 

Category 

0 

1 

2 

3 

4 

5 

Total 

TES 

36 

age 

median= 

20.50 

mean= 

23.00 

sd= 

6.69 

6 

TES 

37 

gender 

6 

TES 

38 

pay  grade 

median= 

muw 

mean= 

#DIWQ! 

sd= 

#DIV/QJ 

6 

TES 

39 

branch  of  service 

median= 

#NUM! 

mean= 

#DIW0! 

sd= 

#DIV/QJ 

6 

TES 

40 

years  enlisted  service 

median= 

1.00 

mearp 

1.00 

sd= 

- 

6 

TES 

41 

years  commissioned  servcie 

median= 

mean= 

- 

sd= 

- 

6 

TES 

42 

designator.  MOS.  or  rating 

median^ 

@8.00 

mean= 

@8.00 

sd= 

#OIV/0! 

6 

TES 

43 

level  of  education  (1-5) 

17% 

a 

33% 

6 

TES 

44 

current  assignment 

3 

TES 

45a 

special  sc  hods:  Ranger 

100% 

6 

TES 

45b 

special  sc  hods:  Pathfinder 

100% 

6 

TES 

45c 

special  sc  hod  s:  El  B 

100% 

6 

TES 

45d 

special  sc  hods:  Air  Assault 

100% 

6 

TES 

45e 

special  sc  hods:  NCO  Course 

100% 

6 

TES 

45f 

special  sc  hods:  SNCO  Course 

100% 

6 

TES 

TES 

45g 

45h 

special  sc  hods:  Computer  Training 
special  sc  hods:  HALO 

100% 

100% 

6 

6 

0 

0 

0 
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=  80%  or  more 
=  50%  to  80% 


Eval 

Category 

Device  used  this  trial 

Device  used  previously 

HDE 

1 

location 

HDE 

2 

date 

HDE 

3 

description  of  scenario 

HDE 

4 

description  of  features 

HDE 

5a 

#texi  messages  senl  (attempted) 

HDE 

5b 

#texl  messages  sen!  (successfii) 

HDE 

6 

satisfaction  with  text  messages  sent 

HDE 

7a 

#  text  messages  received  (attempted) 

HDE 

7b 

#  text  messages  received  [successful) 

HDE 

8 

satisfaction  with  text  messages  received 

HDE 

9 

satisfaction  with  comm  and  network 

HDE 

10 

satisfaction  with  reliability  of  comm  and  network 

HDE 

11 

satisfaction  with  still  images 

HDE 

12a 

#  moving  videos  viewed  (attempted) 

HDE 

12b 

#  moving  videos  viewed  (successful) 

HDE 

13 

satisfaction  with  quality  of  moving  video 

HDE 

14a 

#  audio  files  listened  to  (attempted) 

HDE 

14b 

#  audio  files  listened  b  (successful) 

HDE 

15 

satisfaction  with  quality  of  audo  files 

HDE 

16a 

#  of  uses  of  mapping  functions  (attempted) 

HDE 

16b 

#  of  uses  of  mapping  functions  (successfd) 

HDE 

17 

satisfaction  with  mapping  functions 

HDE 

18 

satisfaction  with  £A  tods 

HDE 

19a 

#  of  uses  of  slyf  us  (attempted) 

HDE 

19b 

#  of  uses  of  stylus  (successfd) 

HDE 

20 

satisfaction  with  stylus 

HDE 

21a 

#  of  uses  of  buttons  (attempted) 

HDE 

21b 

#  of  uses  of  buttons  (successful) 

HDE 

22 

satisfaction  with  buttons 

HDE 

23 

satisfaction  with  one  handed  operation 

HDE 

24 

satisfaction  with  removing  device  to  use  it 

HDE 

25 

satisfaction  withstowng  device 

HDE 

26 

ability  lo  use  personal  weapon 

HDE 

27 

ability  lo  read  the  screen 

HDE 

28 

overall  satisfaction  vuth  reliability 

HDE 

29 

overall  satisfaction  with  ruggedness 

HDE 

30 

overall  satisfaction  with  system 

HDE 

31 

impression  of  ease  of  learning  and  use 

HDE 

32 

free  comments 

mean= 

mean= 


mean= 

mean= 


25% 


25%  25% 


25% 


Total 

6 

6 

0 

0 

0 

0 


4.00 

4.00 

sd= 

sd= 

1.41  4 

1.41  4 

1 

100% 

1  * 

4.25 

4.25 

sd= 

sd= 

0.96  4 

0.96  4 

25% 

4 

100% 

i  4 

50% 

25%  4 

25% 

50% 

4 

3.25 

sd= 

3.20  4 

3.25 

sd= 

3.20  4 

4 

3.50 

sd= 

3.51  4 

3.50 

sd= 

3.51  4 

4 

3.75 

sd= 

0.96  4 

3.75 

sd= 

0.96  4 

25% 

mm 

4 

25% 

25% 

25%  4 

25.00 

sd= 

3336  4 

25.00 

sd= 

3336  4 

25% 

4 

0.25 

sd= 

0.50  4 

0.25 

sd= 

0.50  4 
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=  80%  or  more 
=  50%  to  80% 


Eval 

Category 

Device  used  this  trial 

Device  used  previously 

HDE 

1 

location 

HDE 

2 

date 

HDE 

3 

description  of  scenario 

HDE 

4 

description  of  features 

HDE 

5a 

#texl  messages  serif  (attempted) 

HDE 

5b 

#  text  messages  sent  (success*  J) 

HDE 

6 

satisfaction  with  text  messages  sent 

HDE 

7a 

#texl  messages  received  (attempted) 

HDE 

7b 

#  text  messages  received  [successful) 

HDE 

8 

satisfaction  with  text  messages  received 

HDE 

9 

satisfaction  with  comm  and  network 

HDE 

10 

satisfaction  with  reliability  of  comm  and  network 

HDE 

11 

satisfaction  with  still  images 

HDE 

12a 

#  moving  videos  viewed  (attempted) 

HDE 

12b 

#  movi  ng  vi  deos  vi  ewed  [successful )  | 

HDE 

13 

satisfaction  with  quality  of  moving  video 

HDE 

14a 

#  audi  o  til  es  I  is  tened  to  (attempted)  1 

HDE 

14b 

#  audi  o  fil  es  lis  tened  to  (successful )  | 

HDE 

15 

satisfaction  with  quality  of  audo  files 

HDE 

16a 

#  of  uses  of  mapping  functions  (attempted) 

HDE 

16b 

#  of  uses  of  mapping  functions  (successfd) 

HDE 

17 

satisfaction  with  mapping  functions 

HDE 

18 

satisfaction  with  £A  tods 

HDE 

19a 

#  of  uses  of  slyt  us  (attempted) 

HDE 

19b 

#  of  uses  of  stylus  (successfd) 

HDE 

20 

satisfaction  with  stylus 

HDE 

21a 

#  of  uses  of  buttons  (attempted) 

HDE 

21b 

#  of  uses  of  buttons  (successful) 

HDE 

22 

satisfaction  with  buttons 

HDE 

23 

satisfaction  with  one  handed  operation 

HDE 

24 

satisfaction  with  removing  device  to  use  it 

HDE 

25 

satisfaction  withstowng  device 

HDE 

26 

ability  to  use  personal  weapon 

HDE 

27 

ability  to  read  the  screen 

HDE 

28 

overall  satisfaction  with  reliability 

HDE 

29 

overall  satisfaction  with  ruggedness 

HDE 

30 

overall  satisfaction  with  system 

HDE 

31 

impression  of  ease  of  learning  and  use 

HDE 

32 

free  comments 

RPDA-B 

nta 

no 

0 


yeo 

1 


Total 


median= 

100 

mean= 

2.67 

sd= 

2  89 

3 

median= 

100 

mean= 

2.67 

sd= 

2  89 

3 

33% 

SB 

3 

median= 

1.00 

mean= 

2.67 

sd= 

289 

3 

median= 

1.00 

mean= 

2.67 

sd= 

289 

3 

33% 

ms 

3 

33% 

3 

1 

li)M 

3 

33% 

33% 

33% 

3 

medfan= 

mean= 

0.33 

sd= 

0.58 

3 

median= 

mean= 

0.33 

sd= 

0.58 

3 

67% 

33% 

3 

median^ 

- 

mean= 

0.33 

sd= 

0  58 

3 

median^ 

mean= 

0.33 

sd= 

0  58 

3 

mm 

33% 

3 

median= 

3.00 

mean= 

2.67 

sd= 

0.58 

3 

medan= 

3.00 

mean= 

2.67 

sd= 

058 

3 

33% 

33% 

33% 

3 

33% 

33% 

33% 

3 

median^ 

7.00 

mean= 

20.00 

sd= 

26.06 

3 

median= 

7.00 

mean= 

6.67 

sd= 

3.51 

3 

33% 

ESS 

3 

median= 

- 

mean= 

* 

sd= 

- 

3 

median^ 

mean= 

- 

sd= 

- 

3 

100% 

3 

|  33% 

33% 

33% 

3 

67% 

33% 

3 

67% 

33% 

3 

33% 

33% 

33% 

3 

33% 

33% 

33% 

3 

33% 

33% 

33% 

3 

67%J 

33% 

3 

33% 

33% 

33% 

3 

1 

100%  1 

3 

median= 

#NUM! 

mean= 

#DIW0! 

sd= 

fOV/ffl 

3 
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5.  USABILITY  TEST  DATA 

The  observations  recorded  using  Noldus  Observer  were  summarized 
using  Elementary  Statistics,  Total  Durations,  and  then  exported  to  Microsoft 
Excel.  The  tables  below  show  the  data  in  Excel,  formatted  for  clarity.  The  Scored 
By  column  indicates  the  coder  who  scored  that  video.  When  both  coders  scored 
the  same  file,  the  REV  (reviewed)  column  was  created  by  averaging  the  two 
observations.  If  the  two  observations  were  off  by  more  than  ten  seconds,  the 
original  video  was  reviewed  to  see  if  one  coder  had  made  an  error  or 
misinterpreted  the  scoring  criteria.  In  that  case,  the  REV  column  contains  the 
more  accurate  score  based  on  the  later  review. 


Participant  ID 

501 

502 

503 

503 

503 

504 

504 
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APPENDIX  L 


MORS-TISDALE  PRESENTATION 


This  thesis  was  the  winning  entry  in  the  Military  Operations  Research 
Society’s  (MORS)  -  Tisdale  Graduate  Research  Award  competition.  The 
following  presentation  was  presented  to  the  faculty  of  the  NPS  Operations 
Research  Department  for  the  competition  on  September  6,  2005. 


Put  this  slide  up  immediately  and  leave  up  for 
introduction.  Go  to  next  slide  automatically  when 
speaker  gets  to  that  point. 

"Good  afternoon,  I  am  Captain  Koichi  Takagi  and  ! 
am  honored  to  be  here  in  front  of  you.  " 

[NEXT  SLIDE] 


Background 

Applied  Wcirlighfer  Ergonomics  Canter 
Approach 
Case  Sludy 
Conclusions 

Queslioa  and  Answer  period 

Slide  #2 

“I  will  present  an  overview  of  my  thesis,  and  briefly 
demonstrate  some  of  the  capabilities  of  our 
laboratory.  I  have  a  lot  of  backup  material  on  hand, 
so  at  the  end,  we  can  go  into  more  detail  about  any 
of  the  topics  " 

[PAUSE  -  NEXT  SLIDE] 


technical  note: 

AWE  Blue:  Red  0,  Green  114,  Blue  188 
Presentation  font:  TW  Cen  MT  Condensed 
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Slide  #3 
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Slide  #4 
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[NEXT  SLIDE] 


[0:35] 


[*  from  start  of  refresher  qtr,  8  July  2003,  to  1500  on 
Sep  6;-)] 
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Background 


Slide  #  5 

[CLICK  VIDEO  TO  PLAY,  AND  THEN  CLOSE 
WINDOWS  MEDIA  PLAYER  COMPLETELY] 

"End  users  bear  the  burden  of  poor  design.  Usability 
testing  must  involve  representative  participants 
under  realistic  conditions.  There  is  a  human  factors 
gap  in  the  T&E  process.  This  video  shows  British 
Royal  Marines  landing  in  Basra  using  equipment 
that  may  not  have  been  tested  by  the  right  people 
under  the  right  conditions." 

Various  military  agencies  are  currently  fielding  a 
number  of  devices  known  collectively  as 
Ruggedized  Personal  Digital  Assistants,  or  RPDAs, 


Ruggerfiieri  Personal  Digital  Assistant  (RPDAs) 

isct) 


Slide  #6 


"...  including  these,  which  we  have  been  testing 
within  the  Tactical  Network  Topology  project. 
RPDAs  have  unique  human  factors  considerations. 


Civilian  PDAs  are  not  designed  to  be  used  while  the 
operator .." 


[NEXT  SLIDE] 


[NEXT  SLIDE] 


Military  RPDA  human  factors  considerations 

1.  Simultaneous  demands  an  alteration 

2.  Other  individual  equipment  and  weapons 

3.  Environ  mental  extremes 


simultaneously  attends  to  mental  and  physical 
stimuli," 


Military  RIP  DA  human  factors  considerations 

1.  Si  ntul tu  n  ea  u  s  d  eiri  a  nd  s  on  u  tl  en  ti  on 
?.  Other  individual  equipment  and  weapons 
3.  E  nvi  r  a  nmental  extre  me  s 


"..  wears  or  employs  other  equipment  and  weapons, 
or ..." 


[NEXT  SLIDE] 


[NEXT  SLIDE] 
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Military  RPDA  human  factors  considerations 

1.  Simultaneous  demands  on  attention 

2.  Other  individual  equipment  and  weapons 

3.  Environmental  extremes 


Slide  #  9 

"...  operates  under  environmental  extremes. 

The  Special  Operations  Command  desired  a  human 
factors  evaluation  of  the  RPDAs  they  have  fielded  in 
Iraq  and  Afghanistan. " 

[NEXT  SLIDE] 


Interested  Partners 
Marine  Corps  Warfighting  Lab 
SOCOM  (various  components) 

U$  Arm/  Special  Operations  Command 
US  Arm/  Rapid  Equipping  Force 
US  Army  PEO  Soldier 
US  Navy  Human  Performance  Center 
US  Forest  Service 
N PS  MOVES  Institute 

HPS  Computer  Science  and  ITM  departments 

Noldus  Information  Technology  Slide#  11 

"We  developed  a  state-of-the-art  behavioral 
observation  laboratory  that  has  generated  serious 
interest  from  a  number  of  different  agencies  and  is 
the  basis  for  much  future  research  at  NPS.  The 
developer  of  the  software  is  using  our  lab  as  a 
model  on  their  website." 

[NEXT  SLIDE] 


Mission  Statement 
To  design  and  establish  a  laboratory 
and  a  methodology 
for  scientifically  evaluating 
the  human  factors  considerations 
of  combat  equipment  used  by  military  personnel 
under  both  controlled  laboratory 
and  realistic  field  conditions. 

Slide#  10 

"At  the  time,  no  DoD  facility  existed  that  was 
equipped  to  specifically  test  military  individual 
equipment  in  the  manner  of  software  usability 
testing. 

My  research  interest  was  to  perform  scientifically 
rigorous  human  factors  evaluations  that  went 
beyond  subjective  observations  and  survey  data. 
There  was  a  need  within  the  DoD  for  a  laboratory 
and  a  reliable  research  methodology  to  do  just  that." 

[NEXT  SLIDE] 


Observing  a  user  navigating 
the  NPS  Python  web  page 


Slide  #12 

Here  in  the  lab,  an  observer  watches  the 
participant’s  computer  screen  broadcast  to  his  own, 
while  watching  through  a  one-way  mirror  and 
recording  everything  to  DVD  files  for  later  analysis." 

[NEXT  SLIDE] 
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52”  stereo  plasma  display 


Slide  #13 


Tnrtkal  scenario 

(IMC  Combat  Decision  Range  #37} 


Slide  #14 


"Your  handouts  give  more  detail  about  the  lab 
equipment. 

Aside  from  testing  PC-based  software,  we  can 
immerse  the  participants" 

[NEXT  SLIDE] 


...in  a  multimedia  tactical  scenario..." 

[PLAY  video  clip:  14  participant  watching  screen] 

"...  the  order  of  march  is  a  HMMWV  with  a  .50  cal, 
two  trucks,  and  a  HMMWV  in  the  rear." 


[NEXT  SLIDE  IMMEDIATELY  WHEN  VIDEO  ENDS] 


Two  fixed;  robotic  digital  video  cameras 


Portable  and  wireless  digital  video  cameras 


Slide  #  15 


Slide#  16 


We  record  everything  they  do  from  front  and  side 
views..." 


[NEXT  SLIDE] 


"and  can  record  their  actions  using  a  handheld 
device." 

[NEXT  SLIDE] 
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Mobile  Device  Camera  demo 


Slide#  17 

[  switch  to  live  feed  from  Mobile  Device  Camera:  S-video  input  on 
table  ] 

[begin  demonstration  of  actual  equipment] 

"Up  on  stage  here  with  me  we  have  a  Marine  in  typical  combat  dress. 

I  have  built  battery  packs  for  the  equipment  so  that  a  researcher  can 
follow  a  user  in  the  field,  recording  how  they  operate  the  equipment 
and  the  context.  We  are  seeing  exactly  what  he  does  on  the  screen  of 
his  RPDA.  All  the  necessary  equipment  fits  into  three  waterproof 
transit  cases.  An  actual  field  study  could  validate  the  results  obtained 
from  a  controlled  laboratory  experiment." 


Noldus  Observer  software 


Slide#  18 

"We  bring  all  these  digital  videos  back  into  the 
Noldus  Observer  software,  where  observed 
behaviors  are  manually  coded  into  data  files." 

[NEXT  SLIDE] 


[NEXT  SLIDE] 


[pass  out  lab  manuals] 


Video  control  station  hardware 


There  are  a  lot  of  hardware  and  software 
components  to  the  system  to  install,  " 

[NEXT  SLIDE] 

[including  a  digital  video  mixer,  event  coding 
keyboard,  digital  camera  controller] 


Professional  recording  equipment 


...operate,  and  maintain.  So  that  this  lab  will 
continue  to  be  a  valuable  resource  to  the 
department, " 


[NEXT  SLIDE] 


[Video  mix  monitors,  mic  and  speaker  amps,  the 
wireless  camera  and  microphone  receivers,  and  the 
computer  workstation  and  its  associated  software.] 
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Slide  #21 


"...I’ve  written  a  Users  Guide,  which  distills 
hundreds  of  pages  of  documentation  " 


Users  guide 


Slide  #22 


"  and  usability  references  into  a  focused,  stand¬ 
alone  tutorial." 


[NEXT  SLIDE] 


[NEXT  SLIDE] 


Approach  for  usability  testing 


Approach 


1.  Definition  of  need 
1.  User  identification 

3.  Focos  groups  &  interviews 

4.  Laboratory  and  field  testing 

5.  Pre-  and  post-operation  surveys 
A.  Analysis  and  final  report 


Slide  #  23 

[4:30] 

"I  also  wanted  to  pave  the  way  for  future  research  by 
devising  and  validating..." 

[NEXT  SLIDE] 


Slide  #24 

"a  start-to-finish  methodology  for  performing  a 
usability  study  of  military  individual  equipment.  I 
used  qualitative  and  quantitative  means  to  dissect 
the  problem  of  human  factors  usability  testing." 

[NEXT  SLIDE] 

(My  thesis  details  my  experimental  design,  data 
collection  method,  analytical  procedure,  lessons 
learned  and  my  recommendations  for  future 
researchers.) 
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Case  Study 


Slide  #25 

"To  validate  this  laboratory  and  procedure,  I 
conducted  a  small  case  study  comparing  three 
different  RPDAs. " 

[NEXT  SLIDE] 


Participants 

Should  be  representative  ot  intended  population 
Must  be  available  as  test  participants 
Between-subjects  design  due  to  availability  of 
devices  and  participants  (never  all  at  once) 


Slide  #27 

"Soldiers  and  Marines  at  DLI  brought  to  NPS  in 
small  groups.  { A  between-subjects  experimental 
design  will  usually  be  appropriate  because  neither 
the  different  devices  nor  the  participants  will  be 
available  for  the  whole  study  }" 

[NEXT  SLIDE] 


Case  Study 
Three  devices: 


RPDA-A  RPDA-B  RPDA-D 


Slide  #  26 

"The  case  study  is  a  proof-of-concept  and  was  not 
expected  to  be  conclusive.  This  study  is  continuing, 
to  achieve  the  degree  of  test  significance  that 
SOCOM  desires" 

[NEXT  SLIDE] 


Pre-operation  survey  results 
64%  have  a  cell  phone  (5  hr/wk  median  talk  time) 

50%  have  never  operated  PDA  before 
54%  have  a  computer  at  home 
median  age  20  yrs  old 
84  directly  from  boot  camp 

Slide  #  28 

"The  pre-operation  survey  has  45  questions  that 
attempt  to  capture  the  participant's  technology 
experience  and  their  military  background." 

[5:00] 

[NEXT  SLIDE] 
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Tactical  scenario 


Slide  #  29 

"The  participants  went  through  a  30-minute  tactical 
scenario,  during  which  they  performed..." 


[NEXT  SLIDE] 


Tactical  tasks 

1.  Perform  a  communications  check 

2.  Plot  a  route 

3.  Determine  three  map  coordinates 

4.  Determine  the  bearing  and  range  from  one  point  to 
another 

5.  Send  a  message 

6.  Receive  a  message 

7.  Retrieve  a  previous  message  from  storage 


Slide  #  30 


"...various  communications  and  navigation  tasks. 
The  scenario  drove  them  to  perform  these  tasks 
using  conventional  means" 

[NEXT  SLIDE] 


Tactical  scenario  script 


Slide  #31 

"...(such  as  with  a  paper  map),  to  provide  a  baseline 
measure  of  their  abilities.  Later,  they  performed 
similar  tasks  using  the  RPDA." 

[Pause,  NEXT  SLIDE] 


Laboratory  observations 


Slide  #32 

[play  33  correct  rpda  read  grid] 

"If  the  quantitative  results  are  not  conclusive,  we 
may  still  be  able  to  make  recommendations  based 
on  our  observations  in  the  lab  and  the  field." 

[NEXT  SLIDE] 
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Interface  errors 


Slide  #33 

[play  33  clicking  interface  error] 

"In  this  segment  it  took  23  screen  taps  to  close  the 
dialog  box 

A  great  advantage  of  digital  video  collection  is  that 
you  can  go  back  to  the  original  data  to  interrogate 
outliers" 


[NEXT  SLIDE] 


Tactical  reaction  to  the  scenario,  pt.  II 


Slide  #35 

[play  36  nonchalant  reaction] 

"some  of  which  are  correlated  to  their  survey 
responses" 

[NEXT  SLIDE] 


Tactical  reaction  to  the  scenario,  pt.  I 


Slide  #  34 

"We  saw  a  broad  range  of  behaviors  during  the 
experiment" 

[play  35  tactical  reaction] 

[NEXT  SLIDE  AS  SOON  AS  VIDEO  FINISHES] 


Conventional  mapping,  pt.  I  (Marine) 


Slide  #  36 

[play  37  conv  mapping  while  standing] 

"We  saw  a  range  of  techniques  employed.  For 
example,  some  groups  took  different  tactical 
postures" 

[NEXT  SLIDE  AS  SOON  AS  VIDEO  FINISHES] 
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Conventional  mapping,  pt.  II  (Soldier) 


Slide  #37 

[play  38  conv  mapping  spread] 

"  than  others.  One  group  invariably  removed  their 
weapons  when  mapping. 

Some  assessments  were  subjective..." 

[NEXT  SLIDE  AS  SOON  AS  VIDEO  FINISHES] 


Participant  off-task  due  to  confusing  instructions 


Slide  #39 

[play  40  confused] 

"For  a  subjective  measurement,  how  would  you 
know  when  a  tester  is  confused?" 

[MAKE  SURE  AUDIENCE  HEARS  HIM  SAY,  "I'M 
CONFUSED"  -  REPLAY  IF  NECESSARY] 

[NEXT  SLIDE] 


Unresponsive  interface 


Slide  #  38 

[play  39  nonresponsive] 

"We  tried  to  differentiate  between  time-consuming 
tasks,  hardware  issues,  and  users  misunderstanding 
the  task" 

[NEXT  SLIDE] 


Physical  errors  due  to  device  configuration 


Slide  #  40 

[play  41  fumbling  with  cords] 

"This  device  is  tethered  to  networking  equipment 
and  large  batteries  in  the  backpack,  which  might  not 
be  an  optimal  configuration." 

[NEXT  SLIDE] 
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Coded  data  files 


Plot  of  coded  tasks 


118.93  on  lc  count  chk 
128.03  off  lc  conn  chk 

147.26  on  2c  plot  rte ,  /*  often  reshuffling  rifle; 

snitched  pencils ;  kneels  for  task 
387.46  off  2c  plot  rte 

449.96  mistake,  /*  incorrect  brg/mg  values  for  both 
545.76  tactical  posture 
563.43  on  5c  send  msg 

565.13  adidn  posture 
582.00  on  3c  read  grid 
594.66  off  3c  read  grid 

596.90  slip,  /*  reads  4  coord  grid;  tactical  stress 

613.70  on  5c  send  msg,  /*  had  msg  info  rgilagred 

636.90  off  5c  send  msg 

734.56  on  6c  rev  msg,  /*  kneels  to  write  info 

771.70  on  4c  brg  rng,  /*  kneels 

859.13  mistake,  /*  incorrect  distance 


Slide  #41 


Slide  #  42 


"Researchers  code  the  videos  into  raw  data  files  like 
these,  that  describe  every  observed  event." 

[NEXT  SLIDE] 


"One  way  to  look  at  that  data  is  through  this  time- 
step  plot,  which  shows  what  the  participant  was 
doing  throughout  the  scenario." 

[NEXT  SLIDE] 


Inter-rater  reliability 

(onfusion  matrix 


Inter -rater  reliability 

Confusion  matiix 


0.14  25.24 
0  92.06 

5.8  25.33 


5.5  094  6.44 


Task  time  assessments  by  two  different 
researchers  on  the  same  video  file 

Slide  #43 

"To  verify  that  the  measurements  are  objectively 
coded,  we  look  at  inter-rater  reliability." 

[NEXT  SLIDE] 


Slide  #  44 

"We  compare  the  task  times  assigned  by  a  different 
researcher  on  the  vertical  and  horizontal  axes  and 
find  very  good  agreement  in  this  observation" 

[NEXT  SLIDE] 
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Inter-rater  reliability 


Confusion  matrix 

tl  <2  t3  t4  tS  tfi  t7  Error  Total  A 
tl  22.63  ------  0.33  22.96 

t2  -  54.73  -----  0  54.73 

t3  -  -  9.71  -  -  -  -  0  9.71 

t4  -  -  25.1  -  -  -  0.14  25.24 

t5  ...  92.06  -  -  0  92.06 

t6  -  .  .  .  .  19.53  .  5.8  25.33 

t7  -  -----  5.5  0.94  6.44 

Error  0  1.63  1.65  0  0.17  0  0  -  3.45 

T  ot.  EI22.63  56.36  11.36  25.1  92.23  19.53  5.5  7.21  239.92 


Duration  of  agreements:  229.3  sec 

Duration  of  disagreements:  1 0.66  sec 
Index  of  concordance:  0.96 

Cohen's K:  0.94  Slide  #  45 

"There  are  standard  statistics  we  can  use  to  ensure 
we  are  using  objective  measurement  criteria." 

[NEXT  SLIDE] 

[The  index  of  concordance  is  just  the  proportion  of 
agreements  to  total  observed  time. 

Cohen's  kappa  is  a  measure  of  the  proportion  of 
agreement  that  exceeds  that  expected  by  chance: 

K  =  (observed  proportion  -  proportion  expected  by 
chance  )  /  ( 1  -  proportion  expected  by  chance  ) 


Analytical  method 

Adjust  measurements  for  participant  effects 
Analyze  task  times  for  a  device  effect 
Use  parametric  techniques  for  a  richer  model 
Use  non-parametric  techniques  for  insurance  against 
poor  assumptions  and  small  data  sets 


Slide  #47 

"Quantifying  individual  differences  between 
participants  reduces  the  noise  in  the  measurements, 
and  reduces  our  required  sample  size. 

With  a  properly  constructed  experiment,  we  can 
justify  quantitative  analysis  of  human  factors  data." 

[NEXT  SLIDE] 


Conclusions 


Slide  #46 

"There  are  a  few  key  points  I  considered  in  my  case 
study,  and  propose  for  future  studies" 

[NEXT  SLIDE] 


Participant  effect 

Between-participant  variability  is  greater  than 
between-device  variability 

Finding  factors  that  explain  participant  variability: 

Pre-operation  surveys 

Conventional  tasks  to  establish  baselines 

Slide  #48 

"Testable  measures  of  participant  effect,  like  the 
baseline  tasks,  are  preferable  to  surveys,  which  are 
prone  to  a  variety  of  errors." 

[NEXT  SLIDE] 
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Porametiic  analysis. 

Parametric  analysis 

AN  OVA  model; 

AN  OVA  model; 

>  1  !l .  duiv*  *nv(l'  ilnv ,  rpdi] 

VS  \  bJbi]  1  iiEinia  i.-niiftrifein.-H  IaiE.nra.ls  fen-  ■jwu- L  f  L  rul 

llnieir  j  mi  jjy  t  fro  Tukaf 

EatlMtn  ltd. Error  Iomc  Bound  ITpgiar  BduhL 

R _ B  on.fi  JS.  7  -ft.il 

ru»iilu.l.  T  1M49  1«1 

n-n  HS.fi  jg.l  1.19  170.0  >tn 

i-o  -li.o  33.7  -lm.oo  «S.  s 

Slide  #  49 

Slide  #50 

"Here's  a  simple  example  of  analyzing  this  data.  In 

"and  conclude  that  using  RDA-D  is  in  fact  faster  than 

task  5,  we  see  a  difference  between  devices," 

RPDA-A.  We  also  see  that  device  B  is  better  than  A 

[NEXT  SLIDE] 

at  a  slightly  lower  confidence  level." 

[NEXT  SLIDE] 

Nan  parametric  anal  pis 

Subjective:  analysis 

Kruskal-wallis  tesl 

>  <h,| 

KnuUl-Ihllli  ehln^un  ■  e.OWS,  il£  -  2, 

Highlight  video  dips 

P..IU.  -  GOJfll 

Kiuiha-Hillii  ituiilit  (c  -  9_&5ji : 

B  is  neaer  ifrtn  A.  and  D  rs  toner  Bian  A  fsanw  as  AKOVA> 

Posl -operation  survey  results 

IS-PHi;  c«le  tot  nvM;ipfnp?nwni  pm&pfl 

Subjecl  mailer  expert  review 

Slide  #51 

Slide  #52 

‘We  arrive  at  the  same  conclusions  non- 

"On  the  other  hand,  some  users  did  voice  strong 

para  metrically.  However,  when  adjusting  for 

opinions  about  the  devices  in  their  post-operation 

participant  effects,  we  did  rot  find  a  statistically 

surveys  My  research  has  shown  that  perception  can 

significant  difference  between  the  devices  in  this 

influence  the  adoption  of  innovations  more  strongly 

study." 

than  quantitative  performance. 

[NEXT  SLIDE] 

[NEXT  SLIDE] 
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Great  things  about  my  thesis  experience 

Created  a  research  question  exactly  in  line  with  my 
professional  and  academic  interests. 

Faced  few  constraints  in  devising  my  research  plan. 

Was  actively  involved  in  collecting  my  own  data. 

Had  complete  freedom  in  designing  and  establishing 
a  laboratory  specifically  for  my  research. 

Slide  #  54 

"Some  great  things  about  my  thesis  experience  are 
that  I  was  able  to  pursue  a  research  question  exactly 
in  line  with  my  professional  and  academic  interests,  I 
faced  few  constraints  in  devising  my  research  plan,  I 
was  actively  involved  in  collecting  my  own  data,  and 
I  had  complete  freedom  in  designing  and 
establishing  a  laboratory  specifically  for  my 


Slide  #  55 

"...had  to  come  up  with  my  own  research  questions, 
my  own  research  plan,  I  had  to  get  my  own  data, 
and  I  had  to  build  my  own  lab..." 

[NEXT  SLIDE] 


Slide  #  53 


"My  thesis  fills  the  human  factors  gap  in  the  T&E 
process.  There  are  Special  Forces  operators  and 
Marines  in  Iraq  and  Afghanistan  using  all  of  these 
devices  today.  My  thesis  results  will  be  applied  by 
SOCOM  to  improving  current  and  future  RPDAs. 
This  laboratory  will  make  the  Naval  Postgraduate 
School  the  premiere  military  individual  equipment 


mm, 
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